Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -64,10 +64,10 @@ By the end of this guide you will have:
- **Fleet Connectivity Status** (Step.4, v0.7.79+, enhanced in v0.7.85) - read-only daily snapshot of Arc agent connectivity, physical NIC health, Azure Resource Bridge status, and the node-count reconciliation between cluster `reportedProperties.nodes` and Arc-tagged physical machines. *Scheduled daily 05:30 UTC + manual.*
- **Assess Update Readiness** (Step.5) - pre-flight, report-only readiness + blocking-health snapshot, published as JUnit XML. *Manual only.*
- **Sideload Updates** (Step.6, v0.8.7) - **opt-in, off by default, on-prem self-hosted runner/agent required.** Pre-stages solution-update media onto clusters that cannot pull updates from Azure directly (dark / air-gapped fabrics): Robocopy to the cluster import share, remote SHA256 verify, `Add-SolutionUpdate` import, then flip `UpdateSideloaded=True` for the downstream Step.7 apply. Re-entrant state machine driven by a frequent CRON; the multi-hour copy runs in a detached Scheduled Task. *Inert unless `SIDELOAD_UPDATES=true`. See [sideload.md](docs/sideload.md) and [sideload-robocopy.md](docs/sideload-robocopy.md).*
- **Apply Updates** (Step.7; **v0.8.79 break-glass override**: optional `force_immediate_update` / `forceImmediateUpdate` pipeline parameter (defaults `false`, **manual run / queue only**) bypasses the per-cluster `UpdateStartWindow` / `UpdateExclusionsWindow` maintenance-window gate for emergency / out-of-window patching - the override is unreachable from the scheduled `apply-updates-schedule.yml` configuration file and a high-visibility WARNING banner is emitted into the run log when it fires; **v0.8.78 step-summary UX polish**: ``ScheduleBlocked`` / ``SideloadedBlocked`` / ``ExcludedByTag`` outcomes render as JUnit ``<skipped>`` instead of ``<failure>`` so ``dorny/test-reporter`` no longer flips Step.07 RED on by-design gate-respect outcomes - ``HealthCheckBlocked`` deliberately stays a ``<failure>``; the Readiness KPI table now also surfaces **Already Up to Date** and **Not Ready (needs attention before updating)** rows so operators see the full ring breakdown - eligible, started, skipped, blocked, up-to-date, needs-attention - in one place; GHA-only: ``actions/download-artifact@v6 -> @v7`` silences the Node 20 deprecation warning) - apply updates to a single `UpdateRing` wave at a time, with WhatIf / dry-run support. *Manual only by default - **you must add a schedule** that lines up with your cluster `UpdateStartWindow` tags, see [Step 7 - Apply Updates](docs/appendix-pipelines.md#step-7---apply-updates) and [section 8](#8-scheduling-maintenance-windows-and-change-freeze-periods).*
- **Monitor In-Flight Updates** (Step.8, v0.7.90; v0.7.96 surfaces `Status` + deepest `ErrorMessage` columns, adds the `StepError` stuck-step JUnit type for runs that have hit an error inside a step without crossing the long-running threshold, and renders portal-linked Cluster Name / Update Name cells in the markdown summary; **v0.7.98 UX overhaul**: composite `SeverityScore` sort, per-cell `StateIcon` + `StatusIcon`, horizontal chip stack (`STEP-STUCK` / `RUN-STUCK` / `UNRESOLVED` / `RECENT-FAIL`), `CRITICAL / WARN / OK` fleet status badge at the top of the job summary, collapsible `<details>` Verbose Error block per row, and JUnit `<testsuite time="..">` + `<testcase time="..">` populated with real run elapsed seconds). Operational snapshot during an active wave: lists each cluster whose latest update run is `InProgress`, with current step, progress (`completed/total steps`), elapsed duration, the `Status` column (`Success`/`Error`/`InProgress`/...), and the deepest `errorMessage` walked out of the nested ARM `steps[]` tree; flags long-running runs (default >6h) AND step-errored stuck runs as JUnit failures in the Checks tab. *Scheduled 5x/day at 20:00, 22:00, 00:00, 02:00, 04:00 UTC (every 2h across the typical overnight maintenance window) + manual; default cadence is editable in `monitor-updates.yml` (v0.7.92+).*
- **Fleet Update Status** (Step.9, formerly Step.8; v0.7.96 promotes `NeedsAttention` into the **Update Failed** bucket, adds a new **Action Required** bucket for `PreparationFailed`, and folds `PreparationInProgress` into **Update In Progress**; **v0.7.98** populates JUnit `time=` on each `<testcase>` in the `📜 Update Run History and Error Details` testsuite using `DurationMinutes * 60`). Scheduled daily snapshot of fleet update state, surfaced in the Tests tab. Markdown summary's `📜 Update Run History and Error Details` table now also carries portal-linked Cluster Name / Update Name cells plus the deepest-step `ErrorMessage`. *Scheduled daily 06:00 UTC + manual.*
- **Fleet Health Status** (Step.10, formerly Step.9) - scheduled daily snapshot of 24-hour system health-check failures, surfaced in the Tests tab. *Scheduled daily 07:00 UTC + manual.*
- **Apply Updates** (Step.7; **v0.8.81 step-summary polish**: per-host apply-updates step summary now uses the new shared `Get-AzLocalStatusIconMap` host-aware icon helper (fixes literal `:white_check_mark:` text on Azure DevOps step summaries), the Cluster column wraps in a portal deep-link via `Get-AzLocalClusterPortalLink`, and the standing Ctrl-click tip is single-sourced via `Get-AzLocalCtrlClickTip`; **v0.8.79 break-glass override**: optional `force_immediate_update` / `forceImmediateUpdate` pipeline parameter (defaults `false`, **manual run / queue only**) bypasses the per-cluster `UpdateStartWindow` / `UpdateExclusionsWindow` maintenance-window gate for emergency / out-of-window patching - the override is unreachable from the scheduled `apply-updates-schedule.yml` configuration file and a high-visibility WARNING banner is emitted into the run log when it fires; **v0.8.78 step-summary UX polish**: ``ScheduleBlocked`` / ``SideloadedBlocked`` / ``ExcludedByTag`` outcomes render as JUnit ``<skipped>`` instead of ``<failure>`` so ``dorny/test-reporter`` no longer flips Step.07 RED on by-design gate-respect outcomes - ``HealthCheckBlocked`` deliberately stays a ``<failure>``; the Readiness KPI table now also surfaces **Already Up to Date** and **Not Ready (needs attention before updating)** rows so operators see the full ring breakdown - eligible, started, skipped, blocked, up-to-date, needs-attention - in one place; GHA-only: ``actions/download-artifact@v6 -> @v7`` silences the Node 20 deprecation warning) - apply updates to a single `UpdateRing` wave at a time, with WhatIf / dry-run support. *Manual only by default - **you must add a schedule** that lines up with your cluster `UpdateStartWindow` tags, see [Step 7 - Apply Updates](docs/appendix-pipelines.md#step-7---apply-updates) and [section 8](#8-scheduling-maintenance-windows-and-change-freeze-periods).*
- **Monitor In-Flight Updates** (Step.8, v0.7.90; **v0.8.81 host-aware icons**: the State and Current Step Status columns now use the shared `Get-AzLocalStatusIconMap` helper (fixes literal `:large_blue_circle:` / `:hourglass_flowing_sand:` text on Azure DevOps step summaries); v0.7.96 surfaces `Status` + deepest `ErrorMessage` columns, adds the `StepError` stuck-step JUnit type for runs that have hit an error inside a step without crossing the long-running threshold, and renders portal-linked Cluster Name / Update Name cells in the markdown summary; **v0.7.98 UX overhaul**: composite `SeverityScore` sort, per-cell `StateIcon` + `StatusIcon`, horizontal chip stack (`STEP-STUCK` / `RUN-STUCK` / `UNRESOLVED` / `RECENT-FAIL`), `CRITICAL / WARN / OK` fleet status badge at the top of the job summary, collapsible `<details>` Verbose Error block per row, and JUnit `<testsuite time="..">` + `<testcase time="..">` populated with real run elapsed seconds). Operational snapshot during an active wave: lists each cluster whose latest update run is `InProgress`, with current step, progress (`completed/total steps`), elapsed duration, the `Status` column (`Success`/`Error`/`InProgress`/...), and the deepest `errorMessage` walked out of the nested ARM `steps[]` tree; flags long-running runs (default >6h) AND step-errored stuck runs as JUnit failures in the Checks tab. *Scheduled 5x/day at 20:00, 22:00, 00:00, 02:00, 04:00 UTC (every 2h across the typical overnight maintenance window) + manual; default cadence is editable in `monitor-updates.yml` (v0.7.92+).*
- **Fleet Update Status** (Step.9, formerly Step.8; **v0.8.81 host-aware icons**: the Critical Health Status table, the Primary Status table, and the Version Distribution Support column now use the shared `Get-AzLocalStatusIconMap` helper (fixes literal `:green_circle:` / `:yellow_circle:` / `:information_source:` text on Azure DevOps step summaries); v0.7.96 promotes `NeedsAttention` into the **Update Failed** bucket, adds a new **Action Required** bucket for `PreparationFailed`, and folds `PreparationInProgress` into **Update In Progress**; **v0.7.98** populates JUnit `time=` on each `<testcase>` in the `📜 Update Run History and Error Details` testsuite using `DurationMinutes * 60`). Scheduled daily snapshot of fleet update state, surfaced in the Tests tab. Markdown summary's `📜 Update Run History and Error Details` table now also carries portal-linked Cluster Name / Update Name cells plus the deepest-step `ErrorMessage`. *Scheduled daily 06:00 UTC + manual.*
- **Fleet Health Status** (Step.10, formerly Step.9; **v0.8.81**: fixes the KPI counting bug where Healthy + Unhealthy did not sum to Total - the top KPI table is now split into **Cluster Counts** (Total / Healthy / Unhealthy / **Other**) and **Failing Checks Breakdown**, with a new `other_clusters` step output. Detailed Results columns reordered to put `Title` first and add a collapsible `Description` block that surfaces drive/volume-level detail (e.g. file paths from `Microsoft.Health.FaultType.Volume.FileSystem.Corruption.Correctable` warnings); raw fault-type `Name` preserved for grep/triage) - scheduled daily snapshot of 24-hour system health-check failures, surfaced in the Tests tab. *Scheduled daily 07:00 UTC + manual.*
- An end-to-end "ring-based" rollout pattern: Pilot -> Wave2 -> Production, with each ring gated on the previous wave's success.
- **Optional**: a ServiceNow integration that opens deduped incidents for clusters whose run status indicates the module's own retries cannot recover (failures, blocking health checks, sideloaded payload missing) - see [section 7](#7-optional-open-itsm-tickets-for-clusters-needing-operator-action).

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ parameters:
default: false

variables:
GENERATED_AGAINST_MODULE_VERSION: '0.8.80'
GENERATED_AGAINST_MODULE_VERSION: '0.8.81'
REQUIRED_MODULE_VERSION: '${{ parameters.moduleVersion }}'
reportsPath: '$(Build.ArtifactStagingDirectory)/reports'
# v0.8.7 sideload advisor defaults. Override at the pipeline / variable-group
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ variables:
# log if the YAML appears stale - prompting you to refresh via
# Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5.
- name: GENERATED_AGAINST_MODULE_VERSION
value: '0.8.80'
value: '0.8.81'
# Resolution order for the module version pin (leave all unset to install the latest,
# which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable
# 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ variables:
# the version actually installed and to the latest on PSGallery, and emits a warning
# log if the YAML appears stale - prompting you to refresh via
# Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5.
GENERATED_AGAINST_MODULE_VERSION: '0.8.80'
GENERATED_AGAINST_MODULE_VERSION: '0.8.81'
# Resolution order for the module version pin (leave all unset to install the latest,
# which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable
# 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ variables:
# log if the YAML appears stale - prompting you to refresh via
# Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5.
- name: GENERATED_AGAINST_MODULE_VERSION
value: '0.8.80'
value: '0.8.81'
# Resolution order for the module version pin (leave all unset to install the latest,
# which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable
# 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ variables:
# the version actually installed and to the latest on PSGallery, and emits a warning
# log if the YAML appears stale - prompting you to refresh via
# Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5.
GENERATED_AGAINST_MODULE_VERSION: '0.8.80'
GENERATED_AGAINST_MODULE_VERSION: '0.8.81'
# Resolution order for the module version pin (leave all unset to install the latest,
# which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable
# 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ variables:
# the version actually installed and to the latest on PSGallery, and emits a warning
# log if the YAML appears stale - prompting you to refresh via
# Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5.
GENERATED_AGAINST_MODULE_VERSION: '0.8.80'
GENERATED_AGAINST_MODULE_VERSION: '0.8.81'
# Resolution order for the module version pin (leave all unset to install the latest,
# which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable
# 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ variables:
# the version actually installed and to the latest on PSGallery, and emits a warning
# log if the YAML appears stale - prompting you to refresh via
# Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5.
GENERATED_AGAINST_MODULE_VERSION: '0.8.80'
GENERATED_AGAINST_MODULE_VERSION: '0.8.81'
# Resolution order for the module version pin (leave all unset to install the latest,
# which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable
# 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ variables:
# log if the YAML appears stale - prompting you to refresh via
# Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5.
- name: GENERATED_AGAINST_MODULE_VERSION
value: '0.8.80'
value: '0.8.81'
# Resolution order for the module version pin (leave all unset to install the latest,
# which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable
# 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ variables:
# log if the YAML appears stale - prompting you to refresh via
# Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5.
- name: GENERATED_AGAINST_MODULE_VERSION
value: '0.8.80'
value: '0.8.81'
# Resolution order for the module version pin (leave all unset to install the latest,
# which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable
# 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ parameters:
default: ''

variables:
GENERATED_AGAINST_MODULE_VERSION: '0.8.80'
GENERATED_AGAINST_MODULE_VERSION: '0.8.81'
REQUIRED_MODULE_VERSION: '${{ parameters.moduleVersion }}'
reportsPath: '$(Build.ArtifactStagingDirectory)/reports'

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ variables:
# the version actually installed and to the latest on PSGallery, and emits a warning
# log if the YAML appears stale - prompting you to refresh via
# Copy-AzLocalPipelineExample -Update. See Automation-Pipeline-Examples/README.md section 5.
GENERATED_AGAINST_MODULE_VERSION: '0.8.80'
GENERATED_AGAINST_MODULE_VERSION: '0.8.81'
# Resolution order for the module version pin (leave all unset to install the latest,
# which is the default "fix-forward" behaviour): queue-time parameter > pipeline variable
# 'REQUIRED_MODULE_VERSION' overridden at queue time > empty (latest).
Expand Down
Loading
Loading