Skip to content

fix(airgap): imageset alignment, DNS mirror identity, and airgap secrets completion - #163

Closed
butler54 wants to merge 7 commits into
validatedpatterns:mainfrom
butler54:feat/t1-connected-testing
Closed

butler54 wants to merge 7 commits into
validatedpatterns:mainfrom
butler54:feat/t1-connected-testing

Conversation

@butler54

Copy link
Copy Markdown
Collaborator

Summary

Airgap/deployment robustness fixes from the Tranche 1 testing lifecycle — validated on bare-metal Intel TDX (SNO, OCP 4.22.17) through a full connected pattern deployment plus mirror creation and content certification.

What's in here

Commit Change
fix(airgap): align imageset with installer 4.22.17 and trustee-operator 1.2.1 The imageset pinned OCP 4.22.8 while deployed installers run 4.22.17, and trustee-operator 1.2.0 while the pattern pins CSV v1.2.1 — the mirror must contain the exact release the installer consumes and the operator bundles the pattern resolves
feat(secrets): complete the airgap secrets toggle in enable-airgap-secrets.sh The script only uncommented registryCaCert/bootstrap_secrets; the disconnected flow also needs tdxCollateral enabled and intelPcsApiKey disabled (manual PCK provisioning in airgap). tdxCollateral is now marker-wrapped in the template; idempotent
fix(airgap): add defaultChannel overrides for gitops and gpu-operator in imageset oc-mirror v2 fails catalog filtering when the package's catalog default channel is filtered out (gitops defaults to latest, gpu-operator to v26.7) — defaultChannel re-points the default to the pinned channel
fix(airgap): disable CNV guest images in imageset registry.redhat.io hard-rejects :latest for virtio-win ("must be pulled with an explicit version") — a mirror error on every run. The guest images are documented-optional (~2 GB each) and unconsumed (enableCommonBootImageImport: false)
fix(airgap): disable gpu-operator-certified in T1 imageset Matches the disabled gpu-operator subscription in values-baremetal; its nvcr.io related images hit repeated copy timeouts, and nothing in this tranche consumes the bundle
feat(airgap): DNS-based mirror registry identity The mirror is now identified by DNS name (user-jump.int.apac-tech-lab.net:8443) everywhere: cert CN/SAN (DNS + IP), certs.d trust entries for both, oc-mirror target (so generated cluster-resources carry no IPs — nothing IP-based enters the ignition chain), git-server URL reporting, and the itms-manual-mirrors placeholders (previously applied verbatim with the literal MIRROR_REGISTRY_HOST placeholder — a latent bug) are now substituted from MIRROR_REGISTRY at apply time. Lab host IPs are not stable long-term; DNS is maintained through moves

Validation

  • Connected deployment (node-02, 4.22.17): all 15 ArgoCD applications Synced+Healthy, attestation Affirming (12× POST /kbs/v0/attest 200), resource delivery 200s, all CoCo workloads Running under kata-cc, zero-touch PCK registration via cache reuse
  • Mirror creation & content certification: oc-mirror exit 0 after the defaultChannel/CNV fixes; 273 repositories, 100% content match against the connected run's installed versions (operators, charts, release payload digest), on-disk blob hashes verified
  • The DNS-identity changes were exercised end-to-end: cert regenerated with dual SAN, oc-mirror re-run against the DNS name, and all 37 endpoint references in the generated cluster-resources verified DNS-only

Disconnected deployment status (for transparency)

The full disconnected install on our lab is currently blocked by a lab-network infrastructure fault on the path between the cluster node and the registry host (rate-dependent sustained-TCP collapse; both endpoints validated clean; escalated to the lab owners with instrumentation). These changes are the connected-side + mirror-side output of that work; the disconnected run completes once the lab network issue is resolved. Happy to share the diagnosis details if useful.

Testing

  • bash -n on both modified scripts; YAML validated on all modified manifests
  • The enable-airgap-secrets.sh toggle verified idempotent against a fresh template copy (airgap secrets active, intelPcsApiKey disabled, valid YAML output)
  • Render gate: both the connected pattern (pattern.sh make show) and a direct clustergroup 0.9.58 chart render with the full values chain pass; the airgap overlay selection (global.airgap → sharedValueFiles) verified in the rendered output

…or 1.2.1

The imageset pinned OCP 4.22.8 while the deployed installer is 4.22.17,
and trustee-operator 1.2.0 while values-baremetal pins CSV v1.2.1. The
mirror must contain the exact release the installer consumes (K12) and
the operator bundles the pattern resolves.

Version audit: specs/002-multi-tranche-testing/version-audit.md (coco-gsd)
…crets.sh

The script only uncommented registryCaCert and bootstrap_secrets. The
branch-pair contract for disconnected testing also requires tdxCollateral
enabled (offline DCAP collateral) and intelPcsApiKey disabled (manual PCK
provisioning via dcap-pck.py in airgap; the connected key file must not
be required by load-secrets).

- Wrap the tdxCollateral block in AIRGAP-TDX-COLLATERAL markers in
  values-secret.yaml.template (gen-secrets copies the template verbatim,
  so markers flow through to the generated file)
- Uncomment both marker blocks; comment out the intelPcsApiKey entry
- Idempotent; verified against a fresh template copy (YAML-valid output,
  airgap secrets active, intelPcsApiKey disabled)

Contract: specs/002-multi-tranche-testing/contracts/branch-pair.md (coco-gsd)
… in imageset

oc-mirror v2 fails catalog filtering when the package's catalog default
channel is filtered out: openshift-gitops-operator defaults to 'latest'
(we pin gitops-1.22) and gpu-operator-certified defaults to v26.7 (we
pin v26.3). The defaultChannel field re-points the default to the
pinned channel.

Known-failure class recorded as K13 in the T1 mirror checklist (coco-gsd
specs/002-multi-tranche-testing/mirror-known-failures.md).
…t is a hard mirror error

registry.redhat.io rejects :latest for container-native-virtualization/virtio-win
('must be pulled with an explicit version'), which fails oc-mirror on every run.
The three guest images are documented-optional (~2 GB each) and unconsumed:
Phase 26 sets enableCommonBootImageImport: false, so CNV never imports them.
Commenting them out also removes the large-blob timeout flakiness (rhel8/rhel9
guest images hit context-deadline-exceeded on upload).

T1 mirror run evidence: coco-gsd specs/002-multi-tranche-testing/t1-progress.md
@butler54
butler54 requested a review from a team October 28, 2026 02:55
…disabled subscription

The gpu-operator subscription is commented out in values-baremetal.yaml
(NVIDIA bundle mirror issues); nothing in T1 consumes the bundle. The
nvcr.io related images (dcgm, driver) hit repeated copy timeouts across
two resumable retries, blocking mirror certification for content the
disconnected deployment never pulls.

Tranche 2 re-enables the GPU operator at v26.7 (pawelpros cherry-pick),
which mandates its own re-mirror + re-certification per the T022 trigger
logic — at that point the bundle returns to the imageset.
The mirror registry is now identified by its DNS name everywhere:
- Phase 0-0: canonical MIRROR_REGISTRY=user-jump.int.apac-tech-lab.net:8443,
  with the resolved IP recorded as a generation-time fact only
- Phase 0-1b: cert CN/SAN carry the DNS name (IP SAN kept for transition)
- Phase 0-1h: certs.d trust entries for both name and IP
- oc-mirror targets the DNS name so generated cluster-resources (IDMS/ITMS/
  CatalogSources) reference it — no IP enters the ignition chain
- itms-manual-mirrors placeholders substituted from MIRROR_REGISTRY at apply
  time (was applied verbatim with the literal placeholder)
- git-server URLs and repoURL hints use the DNS name

Lab IPs are not stable (the jump host moved subnets on 2026-09-15, breaking
the T013 install through the reserved-VLAN routing); DNS is maintained
through moves and the node's ignition resolver uses it.

Spec: FR-017 (specs/002-multi-tranche-testing, R6)
…ries

ansible-lint's yaml profile (comments.require-starting-space) flags the
commented-out guest-image entries; add the space after the marker.
@butler54
butler54 force-pushed the feat/t1-connected-testing branch from 335561f to a3b9366 Compare October 28, 2026 03:11
@butler54

butler54 commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator Author

Closing in favor of #164, which is a superset of this PR — it contains all the same commits (imageset alignment, DNS identity, airgap secrets toggle, trustee SNP fix) plus the additional fixes from the Quay consolidation work (vault pin update, pattern-ui-catalog removal, Makefile cleanup, runbook rewrite). No changes are lost.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant