chore: prepare Operator v3.16.1 sandbox patch (EN-2723)

[Dav-14]

Summary

 operator / operator-crds
-version: 3.16.0
-appVersion: v3.16.0
+version: 3.16.1
+appVersion: v3.16.1

Prepare the explicitly selected Operator v3.16.1 candidate for OVH sandbox, starting from v3.16.0 and backporting only already-merged #548 and #544. Align operator / operator-crds chart versions and application versions, regenerate the dependency lock, and document migration and GitOps qualification gates.

Release preparation: EN-2723. Credential behavior: EN-2490; sandbox qualification remains EN-2227 / EN-2231. The Manager owns Jira updates; no Jira write was performed here.

Frozen source

Base chore/v3.16.1-baseline is exactly the existing v3.16.0 tag, 5c92b82d3d1c33f6e28f674ab476c589ecc5be33. This dedicated preparation base avoids incorporating #547 or reverting unrelated main changes. The eventual integration/ref-to-tag decision remains open; this is not a release branch authorization.

Upstream change	Upstream merge	Backport
#548, Ledger superuser API / beta.9 dependency	0cdf98803609dd1510ce02bd79e44e23d209378d	265f63e5
#544 / EN-2490, scoped Credentials and existing Secret binding	be3c687a68ba13c8096ca1b5cc943f5c7a0696c8	e3501912

Range-diff confirms the backports preserve the patches (only provenance trailers differ). Excluded: #547 Job node selectors, #545 Ledger cluster-ID preservation, #542 security dependency updates. The excluded security update requires owner assessment before publication; exclusion is not a safety assertion.

Compatibility impacts — publication blocked

The requested patch number does not certify backward compatibility. Existing Credentials are narrowed to superuser=false and exactly five scopes. A Core still emitting superuser tokens is incompatible. Ledger Credentials CRD/controller must support superuser; Regions currently locks Ledger Operator beta.1, which uses god, and is not a valid candidate prerequisite.

#548 also removes LedgerConfiguration fields (coldStorage, dnsEndpoint, receiptSigning, monitoring.traces.sampling), changes DNS representation and removes the cluster-ID default. Kubernetes pruning plus the reconciler's full Cluster-spec replacement can lose configuration. Inventory real LedgerConfiguration/Cluster objects and obtain an approved migration before applying this candidate.

Required gates: linked explicit human approval of these impacts; compatible Ledger operator/CRDs, Connectivity CRDs and scoped-token Core; actual existing-key migration showing applied grants, ingestion/cursor progress and denial of unrelated privileges; authenticated API qualification; exercised recovery stating the Ledger operator/schema tuple. Ready/observedGeneration proves distribution, not applied grants. Rollback widens privileges and is version-skewed.

The release note includes a LedgerCredentialsPending incompatibility diagnostic (source-derived inference, not exercised) and the exact chart/image evidence still needed. Target is OVH sandbox through its owning GitOps path only; no manual cluster deployment. Stack/GitOps owner selection, final Regions pins and immutable artifact digests remain external qualification gates. AWS/prod is excluded.

Schema inventory correction

The release note now contains all 35 removed schema property paths (parents and descendants). monitoring.traces itself remains present; only its sampling subtree is removed. persistence.data.accessMode retains its original type/default. Source comparison corrects broader textual-diff claims. The Infra owner reports zero explicit LedgerConfiguration objects; Settings, private Secret-backed valuesFrom, chart defaults and existing Cluster specs remain unqualified. The shared Flux CI validation contract is also uninspected. These limits do not waive migration/qualification gates.

OVH compatibility hold — Infra checkpoint

The infrastructure owner independently inspected the live OVH Credentials CRD: cluster-scoped v1alpha1, served/storage, with exactly additionalNamespaces, god, scopes, and selector; superuser is absent. Its read-only evidence packet is retained by the Infra lane. This is corroborating owner evidence, not a live inspection performed by the patch owner. Do not pin or qualify 3.16.1 against the current beta.1 operator/CRD tuple.

The Ledger Operator release owner must be confirmed before any external activation or bump. Require independently proven controller image + served Credentials schema + Core credential format, and inventory removed LedgerConfiguration fields against effective OVH values without exposing secrets. The selected candidate retains key-mode Secret references; a request to verify credential formats does not add a bundle migration or new bundle capability. No manual deployment, CR/Secret mutation, GitOps merge/sync/reconciliation, or external pin change is part of this PR.

Review checkpoint

Head 7a760262e8c7baf38ec165a52e8b0d4f1d624d9b completes the Pyroscope Secret-reference migration procedure. The owner reply records exact-head independent doc review and finding disposition. Prior head ccf152b0 passed Tests, Dirty and all five Kubernetes E2E cells in run 37600883537; those results do not transfer to the new head. Current-head CI is pending. No Slack communication; the PR remains draft under the compatibility/publication hold.

Evidence

Before: v3.16.0 lacks #548/#544 and the charts lock operator-crds 3.16.0. After: the selected backports are present; the generated lock selects 3.16.1 and both rendered charts validate. Runtime/test qualification remains incomplete.

• Passed: Helm dependency regeneration; helm lint --strict and helm template for both charts; rendered Operator image and utils version v3.16.1; git diff --check; exclusion/path inventory and backport range-diff.
• Not passed: focused Connectivities/Ledger race tests failed during compilation with no space left on device; no passing tests are claimed.
• Not completed: required pinned Nix just pre-commit could not realize the SDK because local disk space was exhausted. Unit/envtest suite, Kubernetes matrix and exact-head PR CI remain gates.
• Independent Claude Principal Engineer / Product Engineer / SRE review: publication BLOCKED by schema/credential migration and missing runtime evidence. Findings are disclosed in the release note. Document recheck at c9407e31: Principal/Product/SRE PASS for that disclosure only; affected inventory and Pyroscope procedure independently rechecked at 7a760262: Principal/Product/SRE PASS for documentation accuracy only; it does not approve runtime correctness or publication.

Merge Danger

Door: one-way for a deployed CRD/credential migration without separately verified recovery. Blast Radius: stacks. The PR itself only prepares an isolated candidate; deploying it can prune configuration and narrow grants. Human compatibility approval and real migration evidence remain mandatory.

Draft preparation only. No merge, tag, release, publication, deployment, sync, reconciliation or destructive operation is authorized. No build-images / deploy-staging label or Slack review request.

[NumaryBot]

docs/07-Upgrade/03-Operator v3.16.1.md:16

🟡 [minor] Correct and complete the schema migration inventory

The upgrade gate says monitoring.traces is removed, but the head CRD retains traces and removes only traces.sampling (config CRD lines 2120–2144). The diff also removes persistence.coldCache and replaces monitoring.pyroscope.authToken/basicAuthPassword with Secret references, neither of which is listed. An operator using this inventory to prepare migration can overlook pruned profiling credentials/cache configuration while unnecessarily removing valid tracing configuration.

Suggestion: List traces.sampling rather than traces, include coldCache and both Pyroscope credential replacements, and document moving the profiling credentials into referenced Secrets before applying the schema.

[Dav-14]

Addressed the schema migration finding in head 7a760262e8c7baf38ec165a52e8b0d4f1d624d9b.

The notes now list all 35 removed property paths, including persistence.coldCache and the two inline Pyroscope credential fields; tracing retains monitoring.traces and removes only traces.sampling. The Pyroscope procedure explicitly maps authToken to authTokenFrom and basicAuthPassword to basicAuthPasswordFrom, requires non-empty Secret name/key in the Ledger Cluster namespace, retains the existing auth method/basicAuthUser, and prepares the Secret references through the owning GitOps lane before coordinated schema/controller/configuration adoption. It includes delivery, missing-Secret/key, admission/reconciliation and recovery gates without exposing or creating credentials.

Verified against the generated candidate schema and git diff --check. Independent exact-head Principal/Product/SRE recheck passes for documentation accuracy and accepts the in-scope finding as addressed. This is a fallback issue comment, not a resolvable inline review thread. The full candidate remains blocked on compatible Ledger operator/CRD, effective-config migration, real ingestion/denial and recovery evidence, and linked human compatibility approval. Previous-head Tests/Dirty/five Kubernetes E2E passed; current-head CI is queued. The PR remains draft. No Slack, publication or deployment.

[Dav-14]

Current-head CI checkpoint for 7a760262e8c7baf38ec165a52e8b0d4f1d624d9b (run 37608245435):

Tests and Dirty passed. Kubernetes E2E 1.31 and 1.32 passed after one individual retry each for Docker Hub HTTP 500 errors before tests; 1.35 passed initially. Kubernetes 1.34 also passed after its single individual network retry. The run is complete and remains failed solely on the 1.33 cleanup below. No further job retry is planned.

⚠️ Kubernetes 1.33 remains failed and has not been retried: the webhooks assertions and manual cleanup-finalizers step completed, but Chainsaw cleanup reached its five-minute deadline while deleting fixture Database chainsaw-webhooks-webhooks. The separate auto-created chainsaw-webhooks-ledger Database did exist and was patched/deleted successfully. The fixture Database is absent from the manual finalizer-removal list. Logs do not establish why normal deletion failed, so this is not classified as a harmless infrastructure error or a proven release regression.

The independent review confirms the cleanup observation and that this test fixture is unchanged from v3.16.0. Root-cause evidence is still needed before changing the frozen candidate (v3.16.0 + #548/#544 only). The PR remains draft; runtime compatibility, migration, recovery and human approval gates remain open.