From 471e873cf0c96810c3b2295e431f8b5bc5b91b01 Mon Sep 17 00:00:00 2001 From: David Ragot <35502263+Dav-14@users.noreply.github.com> Date: Wed, 7 Oct 2026 11:20:57 +0200 Subject: [PATCH 1/7] chore: prepare Operator v3.16.1 sandbox patch EN-2490: prepare chart versions and document migration and publication gates for the selected #548/#544 backport. Helm lint/template passed. Required Nix pre-commit could not complete; Go tests failed during compilation because local disk space was exhausted. --- docs/07-Upgrade/03-Operator v3.16.1.md | 32 ++++++++++++++++++++++++++ helm/crds/Chart.yaml | 4 ++-- helm/operator/Chart.lock | 6 ++--- helm/operator/Chart.yaml | 4 ++-- 4 files changed, 39 insertions(+), 7 deletions(-) create mode 100644 docs/07-Upgrade/03-Operator v3.16.1.md diff --git a/docs/07-Upgrade/03-Operator v3.16.1.md b/docs/07-Upgrade/03-Operator v3.16.1.md new file mode 100644 index 00000000..490517ea --- /dev/null +++ b/docs/07-Upgrade/03-Operator v3.16.1.md @@ -0,0 +1,32 @@ +# Operator v3.16.1 candidate + +This targeted OVH sandbox candidate starts from Operator v3.16.0 (`5c92b82d3d1c33f6e28f674ab476c589ecc5be33`) and backports only two changes already merged on main: + +| Change | Upstream commit | Candidate commit | +| --- | --- | --- | +| #548: align Credentials with the Ledger superuser API and beta.9 API dependency | `0cdf98803609dd1510ce02bd79e44e23d209378d` | `265f63e5` | +| #544 / EN-2490: reconcile scoped Connectivity credentials and preserve Secret binding | `be3c687a68ba13c8096ca1b5cc943f5c7a0696c8` | `e3501912` | + +The operator and operator-crds charts both become `3.16.1`, with application version `v3.16.1`. The image references intended for a separately authorized publication are `ghcr.io/formancehq/operator:v3.16.1` and `ghcr.io/formancehq/operator-utils:v3.16.1`, including the existing scratch variants and amd64/arm64 manifests. Neither these images nor the charts are published by this preparation PR. Record their immutable digests and build/source provenance before selecting the Regions pin. + +This candidate excludes main's #547 (Job node selectors), #545 (Ledger cluster-ID preservation), and #542 (deployment/operator and operator-utils security dependency updates). It therefore does not deliver those changes. Assess the excluded security update with its owner before authorizing publication; this is not an assertion that the previous dependency set is safe. + +## Compatibility and migration gates + +This is a behavior and integration-contract change despite the requested patch version. An existing superuser credential is narrowed to five scopes; a Core emitting superuser tokens becomes incompatible. The Ledger API dependency and generated LedgerConfiguration schemas also change. Human approval must explicitly cover these impacts before integration or publication; the selected version number does not certify compatibility. + +1. Install a compatible Ledger operator and Credentials CRD supporting `spec.superuser`, before deploying this Stack Operator. The source API dependency is Ledger beta.9 commit `4fe8ed8c07726da05eb678591e0e4960a821d919`. Regions main currently locks ledger-operator `3.0.0-beta.1`, whose Credentials API uses `spec.god`; that historical pin is not compatible evidence. The Helm lane owns the final compatible chart pin and corresponding runtime image. +2. Install compatible Connectivity CRDs supporting `spec.auth.keyIdSecretKeyRef` and `spec.auth.secretKeyRef`. Bind `key-id` and `seed.hex` from the same Ledger-distributed Secret, with subject `connectivity`. No bundle or derived signing Secret is introduced. +3. Deploy and verify a compatible Core emitting `superuser=false` and the fixed five required scopes before narrowing an existing registered key. Select and record the exact Core/Connectivity operator revisions, images and protocol-compatible Ledger tuple; beta.10 adoption is a separate owned change, not implied by this patch. +4. Exercise existing-credential migration against an actual Ledger. Preserve key ID, Secret identity and namespace isolation. Verify the applied five-scope grant, continued source ingestion, durable cursor progress and denial of unrelated privileges. Credentials Ready plus observedGeneration proves distribution/spec observation; it does not acknowledge applied Ledger grants. +5. Validate the stack Auth issuer and authenticated API access, including allowed and denied tokens. During Auth transitions, route exposure must wait for the delegated Deployment/Service rollout proof. Without a Stack Auth module the existing contract is unauthenticated; explicitly select the intended sandbox configuration. + +Rolling back to v3.16.0 restores the previous superuser grants on reconciliation and the old Ledger API behavior. Treat this as privilege widening with a version-skew risk, requiring explicit operational approval and an exercised recovery plan. Never infer safe rollback from Helm readiness. + +## Preparation branch and delivery evidence + +The preparation PR targets `chore/v3.16.1-baseline`, anchored at the existing v3.16.0 tag. It is not a PR to main: merging the selected backport into main would not remove #547 and cannot produce this frozen patch. The baseline is a review anchor, not an authorized release. The eventual source/ref to tag requires a separate integration decision. No `build-images` or `deploy-staging` label is part of preparation. + +The target is OVH sandbox. Its actual Stack, GitOps source and effective context remain with the infrastructure lane to verify; AWS hosting/production is excluded. + +Track the retained credential contract through EN-2490 and the sandbox qualification through EN-2227 / EN-2231. This PR prepares a candidate only. Required evidence remains: pre-commit and relevant tests on the final tree, exact-head CI, independent Principal Engineer/Product Engineer/SRE review, code-owner approval, explicitly linked approval of the compatibility impacts, release authorization, published artifact digests, aligned Regions locks, and runtime qualification. No merge, tag, release, publication, deployment or environment synchronization is authorized by this document. diff --git a/helm/crds/Chart.yaml b/helm/crds/Chart.yaml index 0160257f..1ff1befa 100644 --- a/helm/crds/Chart.yaml +++ b/helm/crds/Chart.yaml @@ -13,9 +13,9 @@ type: application # This is the chart version. This version number should be incremented each time you make changes # to the chart and its templates, including the app version. # Versions are expected to follow Semantic Versioning (https://semver.org/) -version: "3.16.0" +version: "3.16.1" # This is the version number of the application being deployed. This version number should be # incremented each time you make changes to the application. Versions are not expected to # follow Semantic Versioning. They should reflect the version the application is using. # It is recommended to use it with quotes. -appVersion: "v3.16.0" +appVersion: "v3.16.1" diff --git a/helm/operator/Chart.lock b/helm/operator/Chart.lock index 6843cb5c..1969359c 100644 --- a/helm/operator/Chart.lock +++ b/helm/operator/Chart.lock @@ -1,6 +1,6 @@ dependencies: - name: operator-crds repository: file://../crds - version: 3.16.0 -digest: sha256:8e8ac6dc76d239a75ea7c2b416d75540855a2ded9aff79e60e1d1662c9267a5e -generated: "2026-09-18T15:42:07.770801+02:00" + version: 3.16.1 +digest: sha256:006ccb315031d902806e1d8479171f1def6848cfa866a72768ff75192beafbc7 +generated: "2026-10-07T11:18:50.016649+02:00" diff --git a/helm/operator/Chart.yaml b/helm/operator/Chart.yaml index 2352e6d8..57cd086b 100644 --- a/helm/operator/Chart.yaml +++ b/helm/operator/Chart.yaml @@ -13,12 +13,12 @@ type: application # This is the chart version. This version number should be incremented each time you make changes # to the chart and its templates, including the app version. # Versions are expected to follow Semantic Versioning (https://semver.org/) -version: "3.16.0" +version: "3.16.1" # This is the version number of the application being deployed. This version number should be # incremented each time you make changes to the application. Versions are not expected to # follow Semantic Versioning. They should reflect the version the application is using. # It is recommended to use it with quotes. -appVersion: "v3.16.0" +appVersion: "v3.16.1" dependencies: - name: operator-crds version: "3.X" From 50856c5cc3297342f0672f8f8bfd3f9d32a6041b Mon Sep 17 00:00:00 2001 From: David Ragot <35502263+Dav-14@users.noreply.github.com> Date: Wed, 7 Oct 2026 11:21:23 +0200 Subject: [PATCH 2/7] docs: disclose Ledger schema and GitOps gates for v3.16.1 --- docs/07-Upgrade/03-Operator v3.16.1.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/07-Upgrade/03-Operator v3.16.1.md b/docs/07-Upgrade/03-Operator v3.16.1.md index 490517ea..b297c6a5 100644 --- a/docs/07-Upgrade/03-Operator v3.16.1.md +++ b/docs/07-Upgrade/03-Operator v3.16.1.md @@ -13,7 +13,7 @@ This candidate excludes main's #547 (Job node selectors), #545 (Ledger cluster-I ## Compatibility and migration gates -This is a behavior and integration-contract change despite the requested patch version. An existing superuser credential is narrowed to five scopes; a Core emitting superuser tokens becomes incompatible. The Ledger API dependency and generated LedgerConfiguration schemas also change. Human approval must explicitly cover these impacts before integration or publication; the selected version number does not certify compatibility. +This is a behavior and integration-contract change despite the requested patch version. An existing superuser credential is narrowed to five scopes; a Core emitting superuser tokens becomes incompatible. The Ledger API dependency and generated LedgerConfiguration schemas also change: `coldStorage`, `dnsEndpoint`, `receiptSigning`, and `monitoring.traces` are removed; `dnsEndpoints`, `clusterPolicyRevision`, and five `metadataMax*` fields are added; `clusterID` loses its `default` value. Kubernetes can prune removed fields on CRD update. The Ledger reconciler replaces the full delegated Cluster spec from its typed configuration, so reconciliation can also remove fields from existing Clusters. Inventory the actual LedgerConfiguration and Cluster objects and preserve their intended configuration through an approved migration before applying the new CRD or Operator. Human approval must explicitly cover these impacts before integration or publication; the selected version number does not certify compatibility. 1. Install a compatible Ledger operator and Credentials CRD supporting `spec.superuser`, before deploying this Stack Operator. The source API dependency is Ledger beta.9 commit `4fe8ed8c07726da05eb678591e0e4960a821d919`. Regions main currently locks ledger-operator `3.0.0-beta.1`, whose Credentials API uses `spec.god`; that historical pin is not compatible evidence. The Helm lane owns the final compatible chart pin and corresponding runtime image. 2. Install compatible Connectivity CRDs supporting `spec.auth.keyIdSecretKeyRef` and `spec.auth.secretKeyRef`. Bind `key-id` and `seed.hex` from the same Ledger-distributed Secret, with subject `connectivity`. No bundle or derived signing Secret is introduced. @@ -21,12 +21,14 @@ This is a behavior and integration-contract change despite the requested patch v 4. Exercise existing-credential migration against an actual Ledger. Preserve key ID, Secret identity and namespace isolation. Verify the applied five-scope grant, continued source ingestion, durable cursor progress and denial of unrelated privileges. Credentials Ready plus observedGeneration proves distribution/spec observation; it does not acknowledge applied Ledger grants. 5. Validate the stack Auth issuer and authenticated API access, including allowed and denied tokens. During Auth transitions, route exposure must wait for the delegated Deployment/Service rollout proof. Without a Stack Auth module the existing contract is unauthenticated; explicitly select the intended sandbox configuration. -Rolling back to v3.16.0 restores the previous superuser grants on reconciliation and the old Ledger API behavior. Treat this as privilege widening with a version-skew risk, requiring explicit operational approval and an exercised recovery plan. Never infer safe rollback from Helm readiness. +If Connectivity remains in `LedgerCredentialsPending`, inspect the served Credentials CRD and Ledger operator revision first. A legacy CRD can prune `superuser`; the Operator may then repeatedly report an update and remain pending while legacy god grants remain active. This failure signature is inferred from source and must be exercised or rejected by runtime evidence; do not treat an old Ready status as success. + +Rolling back to v3.16.0 restores the previous superuser grants on reconciliation and the old Ledger API behavior. Treat this as privilege widening with a version-skew risk, requiring explicit operational approval and an exercised recovery plan that states the Ledger operator version and the LedgerConfiguration/Cluster schema used before and after recovery. Never infer safe rollback from Helm readiness. ## Preparation branch and delivery evidence The preparation PR targets `chore/v3.16.1-baseline`, anchored at the existing v3.16.0 tag. It is not a PR to main: merging the selected backport into main would not remove #547 and cannot produce this frozen patch. The baseline is a review anchor, not an authorized release. The eventual source/ref to tag requires a separate integration decision. No `build-images` or `deploy-staging` label is part of preparation. -The target is OVH sandbox. Its actual Stack, GitOps source and effective context remain with the infrastructure lane to verify; AWS hosting/production is excluded. +The target is OVH sandbox. Changes are prepared for the owning GitOps path only; no manual deployment to these clusters is authorized, and GitOps merge/reconciliation/sync require separate authorization. Its actual Stack, GitOps source and effective context remain with the infrastructure lane to verify; AWS hosting/production is excluded. Track the retained credential contract through EN-2490 and the sandbox qualification through EN-2227 / EN-2231. This PR prepares a candidate only. Required evidence remains: pre-commit and relevant tests on the final tree, exact-head CI, independent Principal Engineer/Product Engineer/SRE review, code-owner approval, explicitly linked approval of the compatibility impacts, release authorization, published artifact digests, aligned Regions locks, and runtime qualification. No merge, tag, release, publication, deployment or environment synchronization is authorized by this document. From 2f6110709e9f66bd63d6c5e62d84f00b3e3ae883 Mon Sep 17 00:00:00 2001 From: David Ragot <35502263+Dav-14@users.noreply.github.com> Date: Wed, 7 Oct 2026 11:26:22 +0200 Subject: [PATCH 3/7] docs: inventory removed LedgerConfiguration schema paths --- docs/07-Upgrade/03-Operator v3.16.1.md | 50 +++++++++++++++++++++++++- 1 file changed, 49 insertions(+), 1 deletion(-) diff --git a/docs/07-Upgrade/03-Operator v3.16.1.md b/docs/07-Upgrade/03-Operator v3.16.1.md index b297c6a5..facf2742 100644 --- a/docs/07-Upgrade/03-Operator v3.16.1.md +++ b/docs/07-Upgrade/03-Operator v3.16.1.md @@ -13,7 +13,7 @@ This candidate excludes main's #547 (Job node selectors), #545 (Ledger cluster-I ## Compatibility and migration gates -This is a behavior and integration-contract change despite the requested patch version. An existing superuser credential is narrowed to five scopes; a Core emitting superuser tokens becomes incompatible. The Ledger API dependency and generated LedgerConfiguration schemas also change: `coldStorage`, `dnsEndpoint`, `receiptSigning`, and `monitoring.traces` are removed; `dnsEndpoints`, `clusterPolicyRevision`, and five `metadataMax*` fields are added; `clusterID` loses its `default` value. Kubernetes can prune removed fields on CRD update. The Ledger reconciler replaces the full delegated Cluster spec from its typed configuration, so reconciliation can also remove fields from existing Clusters. Inventory the actual LedgerConfiguration and Cluster objects and preserve their intended configuration through an approved migration before applying the new CRD or Operator. Human approval must explicitly cover these impacts before integration or publication; the selected version number does not certify compatibility. +This is a behavior and integration-contract change despite the requested patch version. An existing superuser credential is narrowed to five scopes; a Core emitting superuser tokens becomes incompatible. The Ledger API dependency and generated LedgerConfiguration schemas also change: `coldStorage`, `dnsEndpoint`, `receiptSigning`, and `monitoring.traces.sampling` are removed; `dnsEndpoints`, `clusterPolicyRevision`, and five `metadataMax*` fields are added; `clusterID` loses its `default` value. Kubernetes can prune removed fields when objects pass through the updated schema; replacing the CRD alone is not proof that all stored objects have already changed. The Ledger reconciler replaces the full delegated Cluster spec from its typed configuration, so reconciliation can also remove fields from existing Clusters. Inventory the actual LedgerConfiguration and Cluster objects and preserve their intended configuration through an approved migration before applying the new CRD or Operator. Human approval must explicitly cover these impacts before integration or publication; the selected version number does not certify compatibility. 1. Install a compatible Ledger operator and Credentials CRD supporting `spec.superuser`, before deploying this Stack Operator. The source API dependency is Ledger beta.9 commit `4fe8ed8c07726da05eb678591e0e4960a821d919`. Regions main currently locks ledger-operator `3.0.0-beta.1`, whose Credentials API uses `spec.god`; that historical pin is not compatible evidence. The Helm lane owns the final compatible chart pin and corresponding runtime image. 2. Install compatible Connectivity CRDs supporting `spec.auth.keyIdSecretKeyRef` and `spec.auth.secretKeyRef`. Bind `key-id` and `seed.hex` from the same Ledger-distributed Secret, with subject `connectivity`. No bundle or derived signing Secret is introduced. @@ -25,6 +25,54 @@ If Connectivity remains in `LedgerCredentialsPending`, inspect the served Creden Rolling back to v3.16.0 restores the previous superuser grants on reconciliation and the old Ledger API behavior. Treat this as privilege widening with a version-skew risk, requiring explicit operational approval and an exercised recovery plan that states the Ledger operator version and the LedgerConfiguration/Cluster schema used before and after recovery. Never infer safe rollback from Helm readiness. +## Exact removed schema paths and OVH evidence limits + +Comparing the v1beta1 structural schema in `config/crd/bases/formance.com_ledgerconfigurations.yaml` at v3.16.0 and #548 (`0cdf98803609dd1510ce02bd79e44e23d209378d`) yields the following 35 removed property paths. `[]` marks array items; parent and child paths are listed separately. This inventory describes source schema changes, not effective live configuration. + +```text +spec.cluster.coldStorage +spec.cluster.coldStorage.bucketId +spec.cluster.coldStorage.driver +spec.cluster.coldStorage.path +spec.cluster.coldStorage.s3 +spec.cluster.coldStorage.s3.bucket +spec.cluster.coldStorage.s3.endpoint +spec.cluster.coldStorage.s3.region +spec.cluster.dnsEndpoint +spec.cluster.dnsEndpoint.annotations +spec.cluster.dnsEndpoint.enabled +spec.cluster.dnsEndpoint.endpoints +spec.cluster.dnsEndpoint.endpoints[].dnsName +spec.cluster.dnsEndpoint.endpoints[].providerSpecific +spec.cluster.dnsEndpoint.endpoints[].providerSpecific[].name +spec.cluster.dnsEndpoint.endpoints[].providerSpecific[].value +spec.cluster.dnsEndpoint.endpoints[].recordTTL +spec.cluster.dnsEndpoint.endpoints[].recordType +spec.cluster.dnsEndpoint.endpoints[].targets +spec.cluster.monitoring.pyroscope.authToken +spec.cluster.monitoring.pyroscope.basicAuthPassword +spec.cluster.monitoring.traces.sampling +spec.cluster.monitoring.traces.sampling.enabled +spec.cluster.monitoring.traces.sampling.successRatio +spec.cluster.persistence.coldCache +spec.cluster.persistence.coldCache.accessMode +spec.cluster.persistence.coldCache.hostPath +spec.cluster.persistence.coldCache.hostPath.path +spec.cluster.persistence.coldCache.hostPath.type +spec.cluster.persistence.coldCache.size +spec.cluster.persistence.coldCache.storageClass +spec.cluster.persistence.coldCache.volumeAttributesClassName +spec.cluster.receiptSigning +spec.cluster.receiptSigning.secretKey +spec.cluster.receiptSigning.secretName +``` + +`spec.cluster.monitoring.traces` remains present; only its `sampling` subtree is removed. `spec.cluster.persistence.data.accessMode` retains the same type and `ReadWriteOnce` default. Their apparent removal in a textual diff must not be treated as a removed schema path. The `spec.cluster.clusterID` property remains present but loses its `default: default` value; path comparison alone does not detect that changed default. + +The Infra owner's read-only OVH checkpoint reports zero explicit LedgerConfiguration objects. This does not prove that Settings, private Helm values, chart defaults, future manifests, or existing Ledger Cluster specs do not use removed configuration. Regions consumes a Secret through valuesFrom; no secret content was read or decrypted. Owners must compare authorized redacted effective values and Cluster specs with this schema before approving migration. + +The OVH Flux CI includes a shared `fluxcd.yml` template; its effective validation contract has not been inspected in this lane. No passing render/schema gate is inferred from that include. The Flux writer/grant, exact Stack, compatible published tuple, effective-values render, independent reviews and exercised runtime/recovery remain gates. No Flux edit, branch, push, manual deployment, AWS diff or cluster mutation is performed by this inventory. + ## Preparation branch and delivery evidence The preparation PR targets `chore/v3.16.1-baseline`, anchored at the existing v3.16.0 tag. It is not a PR to main: merging the selected backport into main would not remove #547 and cannot produce this frozen patch. The baseline is a review anchor, not an authorized release. The eventual source/ref to tag requires a separate integration decision. No `build-images` or `deploy-staging` label is part of preparation. From 7ec023776ba103329b20e58eabee153c2b277a85 Mon Sep 17 00:00:00 2001 From: David Ragot <35502263+Dav-14@users.noreply.github.com> Date: Wed, 7 Oct 2026 12:31:39 +0200 Subject: [PATCH 4/7] docs: explain Pyroscope Secret-reference migration --- docs/07-Upgrade/03-Operator v3.16.1.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/docs/07-Upgrade/03-Operator v3.16.1.md b/docs/07-Upgrade/03-Operator v3.16.1.md index facf2742..63e85318 100644 --- a/docs/07-Upgrade/03-Operator v3.16.1.md +++ b/docs/07-Upgrade/03-Operator v3.16.1.md @@ -73,6 +73,25 @@ The Infra owner's read-only OVH checkpoint reports zero explicit LedgerConfigura The OVH Flux CI includes a shared `fluxcd.yml` template; its effective validation contract has not been inspected in this lane. No passing render/schema gate is inferred from that include. The Flux writer/grant, exact Stack, compatible published tuple, effective-values render, independent reviews and exercised runtime/recovery remain gates. No Flux edit, branch, push, manual deployment, AWS diff or cluster mutation is performed by this inventory. +## Pyroscope credential migration before schema adoption + +The new schema replaces inline profiling credentials with Secret references. Each reference requires a non-empty Secret `name` and `key` and resolves in the delegated Ledger Cluster namespace: + +| Removed field under `spec.cluster.monitoring.pyroscope` | Replacement | +| --- | --- | +| `authToken` | `authTokenFrom.name` and `authTokenFrom.key` | +| `basicAuthPassword` | `basicAuthPasswordFrom.name` and `basicAuthPasswordFrom.key` | + +Keep `basicAuthUser` where basic authentication is used; that field remains supported. Retain the existing authentication method rather than enabling both modes as part of this migration. + +1. The configuration owner inventories which profiling authentication mode is used in the approved effective configuration, including Settings, private Helm values and existing Cluster specs. Record only field/reference presence and namespace ownership; never print credential values or copy them into this repository or review comments. +2. Before adopting the new schema, the owning GitOps lane prepares the existing credential material through its approved Secret-management mechanism in each relevant Cluster namespace. Do not remove the old configuration until the compatible Ledger operator can resolve the new references. This preparation document authorizes no Secret creation, decryption or cluster mutation. +3. Prepare the Secret selectors and the matching compatible operator/CRD rollout together. Do not apply the selectors to an old schema that prunes them, or drop the inline fields before the references can be used. Review the effective configuration without secrets and establish the explicit ordering/recovery plan before any separately authorized GitOps integration or reconciliation. +4. During separately authorized qualification, verify that profiling delivery still succeeds and that a missing Secret/key produces an actionable failure. Check that reference-only configuration survives admission and reconciliation without silently disabling profiling. Readiness alone does not establish successful authentication to the profiling service. +5. Recover using the approved operator/schema/configuration tuple if reference resolution or delivery fails. Preserve the original credential material in the approved Secret-management system for that recovery window; do not reconstruct inline credentials from this document or assume a downgrade understands the new selectors. + +This is a required migration procedure for existing profiling users, not evidence that OVH uses these fields or that any credential has been moved. The actual configuration inventory, Secret owner, rollout ordering and exercised recovery remain gates. + ## Preparation branch and delivery evidence The preparation PR targets `chore/v3.16.1-baseline`, anchored at the existing v3.16.0 tag. It is not a PR to main: merging the selected backport into main would not remove #547 and cannot produce this frozen patch. The baseline is a review anchor, not an authorized release. The eventual source/ref to tag requires a separate integration decision. No `build-images` or `deploy-staging` label is part of preparation. From e04e195381d4a02ed1e38573de6a89e1b60d65de Mon Sep 17 00:00:00 2001 From: David Ragot <35502263+Dav-14@users.noreply.github.com> Date: Thu, 8 Oct 2026 18:00:35 +0200 Subject: [PATCH 5/7] docs: align release preparation with main and Ledger beta.10 --- docs/07-Upgrade/03-Operator v3.16.1.md | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-) diff --git a/docs/07-Upgrade/03-Operator v3.16.1.md b/docs/07-Upgrade/03-Operator v3.16.1.md index 63e85318..494eb00b 100644 --- a/docs/07-Upgrade/03-Operator v3.16.1.md +++ b/docs/07-Upgrade/03-Operator v3.16.1.md @@ -1,23 +1,18 @@ # Operator v3.16.1 candidate -This targeted OVH sandbox candidate starts from Operator v3.16.0 (`5c92b82d3d1c33f6e28f674ab476c589ecc5be33`) and backports only two changes already merged on main: - -| Change | Upstream commit | Candidate commit | -| --- | --- | --- | -| #548: align Credentials with the Ledger superuser API and beta.9 API dependency | `0cdf98803609dd1510ce02bd79e44e23d209378d` | `265f63e5` | -| #544 / EN-2490: reconcile scoped Connectivity credentials and preserve Secret binding | `be3c687a68ba13c8096ca1b5cc943f5c7a0696c8` | `e3501912` | +This release-preparation PR is based on `main` at `12eac9b10344d59cae66d626d9cbd97f98736c26`. The scoped Connectivity credential changes (#548 and #544) are already integrated there; this PR adds the version and migration notes rather than backporting their implementation. The operator and operator-crds charts both become `3.16.1`, with application version `v3.16.1`. The image references intended for a separately authorized publication are `ghcr.io/formancehq/operator:v3.16.1` and `ghcr.io/formancehq/operator-utils:v3.16.1`, including the existing scratch variants and amd64/arm64 manifests. Neither these images nor the charts are published by this preparation PR. Record their immutable digests and build/source provenance before selecting the Regions pin. -This candidate excludes main's #547 (Job node selectors), #545 (Ledger cluster-ID preservation), and #542 (deployment/operator and operator-utils security dependency updates). It therefore does not deliver those changes. Assess the excluded security update with its owner before authorizing publication; this is not an assertion that the previous dependency set is safe. +The rebased candidate includes the existing main changes: #547 (Job node selectors), #545 (Ledger cluster-ID preservation), #542 (security dependency updates), and #550 (shared GoReleaser Docker v2 preset), as well as #548/#544. It is no longer the earlier isolated v3.16.0 + #548/#544 candidate. Assess the complete main-based release delta before publication. ## Compatibility and migration gates -This is a behavior and integration-contract change despite the requested patch version. An existing superuser credential is narrowed to five scopes; a Core emitting superuser tokens becomes incompatible. The Ledger API dependency and generated LedgerConfiguration schemas also change: `coldStorage`, `dnsEndpoint`, `receiptSigning`, and `monitoring.traces.sampling` are removed; `dnsEndpoints`, `clusterPolicyRevision`, and five `metadataMax*` fields are added; `clusterID` loses its `default` value. Kubernetes can prune removed fields when objects pass through the updated schema; replacing the CRD alone is not proof that all stored objects have already changed. The Ledger reconciler replaces the full delegated Cluster spec from its typed configuration, so reconciliation can also remove fields from existing Clusters. Inventory the actual LedgerConfiguration and Cluster objects and preserve their intended configuration through an approved migration before applying the new CRD or Operator. Human approval must explicitly cover these impacts before integration or publication; the selected version number does not certify compatibility. +This is a behavior and integration-contract change despite the requested patch version. An existing superuser credential is narrowed to five scopes; a Core emitting superuser tokens becomes incompatible. The Ledger API dependency and generated LedgerConfiguration schemas also change: `coldStorage`, `dnsEndpoint`, `receiptSigning`, and `monitoring.traces.sampling` are removed; `dnsEndpoints`, `clusterPolicyRevision`, and five `metadataMax*` fields are added; `clusterID` loses its schema `default` value; #545 preserves the delegated Ledger operator's generated cluster ID when configuration leaves it unset. Kubernetes can prune removed fields when objects pass through the updated schema; replacing the CRD alone is not proof that all stored objects have already changed. The Ledger reconciler replaces the full delegated Cluster spec from its typed configuration, so reconciliation can also remove fields from existing Clusters. Inventory the actual LedgerConfiguration and Cluster objects and preserve their intended configuration through an approved migration before applying the new CRD or Operator. Human approval must explicitly cover these impacts before integration or publication; the selected version number does not certify compatibility. -1. Install a compatible Ledger operator and Credentials CRD supporting `spec.superuser`, before deploying this Stack Operator. The source API dependency is Ledger beta.9 commit `4fe8ed8c07726da05eb678591e0e4960a821d919`. Regions main currently locks ledger-operator `3.0.0-beta.1`, whose Credentials API uses `spec.god`; that historical pin is not compatible evidence. The Helm lane owns the final compatible chart pin and corresponding runtime image. +1. Install a compatible Ledger operator and Credentials CRD supporting `spec.superuser`, before deploying this Stack Operator. Ledger beta.10 is the selected runtime target. The Stack Operator Go API dependency still points to beta.9 commit `4fe8ed8c07726da05eb678591e0e4960a821d919`; that compile-time pin does not establish the deployed Ledger version or qualify the beta.10 controller/CRD tuple. The last verified OVH Ledger operator/CRD checkpoint was beta.1 with `spec.god`; refresh the target's actual image and served schema before rollout. The Helm lane owns the final compatible chart pin and corresponding runtime image. 2. Install compatible Connectivity CRDs supporting `spec.auth.keyIdSecretKeyRef` and `spec.auth.secretKeyRef`. Bind `key-id` and `seed.hex` from the same Ledger-distributed Secret, with subject `connectivity`. No bundle or derived signing Secret is introduced. -3. Deploy and verify a compatible Core emitting `superuser=false` and the fixed five required scopes before narrowing an existing registered key. Select and record the exact Core/Connectivity operator revisions, images and protocol-compatible Ledger tuple; beta.10 adoption is a separate owned change, not implied by this patch. +3. Deploy and verify a compatible Core emitting `superuser=false` and the fixed five required scopes before narrowing an existing registered key. Select and record the exact Core/Connectivity operator revisions, images and protocol-compatible Ledger tuple; Core beta.2 at `ce2324887f5b5ec4e3c2ec934ac874656d4c5348` requires Discovery protocol 20 and matching reflected descriptors before every Apply; its synced protobuf source matches Ledger beta.10. Qualify the selected beta.10 image, credentials and data compatibility on the exact Stack; this preparation does not authorize a Ledger upgrade. 4. Exercise existing-credential migration against an actual Ledger. Preserve key ID, Secret identity and namespace isolation. Verify the applied five-scope grant, continued source ingestion, durable cursor progress and denial of unrelated privileges. Credentials Ready plus observedGeneration proves distribution/spec observation; it does not acknowledge applied Ledger grants. 5. Validate the stack Auth issuer and authenticated API access, including allowed and denied tokens. During Auth transitions, route exposure must wait for the delegated Deployment/Service rollout proof. Without a Stack Auth module the existing contract is unauthenticated; explicitly select the intended sandbox configuration. @@ -94,7 +89,7 @@ This is a required migration procedure for existing profiling users, not evidenc ## Preparation branch and delivery evidence -The preparation PR targets `chore/v3.16.1-baseline`, anchored at the existing v3.16.0 tag. It is not a PR to main: merging the selected backport into main would not remove #547 and cannot produce this frozen patch. The baseline is a review anchor, not an authorized release. The eventual source/ref to tag requires a separate integration decision. No `build-images` or `deploy-staging` label is part of preparation. +The preparation PR targets `main`. A main-based release includes changes beyond the earlier isolated two-fix candidate; record the final source SHA and complete delta against v3.16.0 before tagging. The earlier baseline branch is historical preparation evidence, not the integration target. No `build-images` or `deploy-staging` label is part of preparation. The target is OVH sandbox. Changes are prepared for the owning GitOps path only; no manual deployment to these clusters is authorized, and GitOps merge/reconciliation/sync require separate authorization. Its actual Stack, GitOps source and effective context remain with the infrastructure lane to verify; AWS hosting/production is excluded. From f9e757dddc8595417e5f7d4551492a18b2480459 Mon Sep 17 00:00:00 2001 From: David Ragot <35502263+Dav-14@users.noreply.github.com> Date: Thu, 8 Oct 2026 18:05:41 +0200 Subject: [PATCH 6/7] docs: remove release upgrade note from preparation PR --- docs/07-Upgrade/03-Operator v3.16.1.md | 96 -------------------------- 1 file changed, 96 deletions(-) delete mode 100644 docs/07-Upgrade/03-Operator v3.16.1.md diff --git a/docs/07-Upgrade/03-Operator v3.16.1.md b/docs/07-Upgrade/03-Operator v3.16.1.md deleted file mode 100644 index 494eb00b..00000000 --- a/docs/07-Upgrade/03-Operator v3.16.1.md +++ /dev/null @@ -1,96 +0,0 @@ -# Operator v3.16.1 candidate - -This release-preparation PR is based on `main` at `12eac9b10344d59cae66d626d9cbd97f98736c26`. The scoped Connectivity credential changes (#548 and #544) are already integrated there; this PR adds the version and migration notes rather than backporting their implementation. - -The operator and operator-crds charts both become `3.16.1`, with application version `v3.16.1`. The image references intended for a separately authorized publication are `ghcr.io/formancehq/operator:v3.16.1` and `ghcr.io/formancehq/operator-utils:v3.16.1`, including the existing scratch variants and amd64/arm64 manifests. Neither these images nor the charts are published by this preparation PR. Record their immutable digests and build/source provenance before selecting the Regions pin. - -The rebased candidate includes the existing main changes: #547 (Job node selectors), #545 (Ledger cluster-ID preservation), #542 (security dependency updates), and #550 (shared GoReleaser Docker v2 preset), as well as #548/#544. It is no longer the earlier isolated v3.16.0 + #548/#544 candidate. Assess the complete main-based release delta before publication. - -## Compatibility and migration gates - -This is a behavior and integration-contract change despite the requested patch version. An existing superuser credential is narrowed to five scopes; a Core emitting superuser tokens becomes incompatible. The Ledger API dependency and generated LedgerConfiguration schemas also change: `coldStorage`, `dnsEndpoint`, `receiptSigning`, and `monitoring.traces.sampling` are removed; `dnsEndpoints`, `clusterPolicyRevision`, and five `metadataMax*` fields are added; `clusterID` loses its schema `default` value; #545 preserves the delegated Ledger operator's generated cluster ID when configuration leaves it unset. Kubernetes can prune removed fields when objects pass through the updated schema; replacing the CRD alone is not proof that all stored objects have already changed. The Ledger reconciler replaces the full delegated Cluster spec from its typed configuration, so reconciliation can also remove fields from existing Clusters. Inventory the actual LedgerConfiguration and Cluster objects and preserve their intended configuration through an approved migration before applying the new CRD or Operator. Human approval must explicitly cover these impacts before integration or publication; the selected version number does not certify compatibility. - -1. Install a compatible Ledger operator and Credentials CRD supporting `spec.superuser`, before deploying this Stack Operator. Ledger beta.10 is the selected runtime target. The Stack Operator Go API dependency still points to beta.9 commit `4fe8ed8c07726da05eb678591e0e4960a821d919`; that compile-time pin does not establish the deployed Ledger version or qualify the beta.10 controller/CRD tuple. The last verified OVH Ledger operator/CRD checkpoint was beta.1 with `spec.god`; refresh the target's actual image and served schema before rollout. The Helm lane owns the final compatible chart pin and corresponding runtime image. -2. Install compatible Connectivity CRDs supporting `spec.auth.keyIdSecretKeyRef` and `spec.auth.secretKeyRef`. Bind `key-id` and `seed.hex` from the same Ledger-distributed Secret, with subject `connectivity`. No bundle or derived signing Secret is introduced. -3. Deploy and verify a compatible Core emitting `superuser=false` and the fixed five required scopes before narrowing an existing registered key. Select and record the exact Core/Connectivity operator revisions, images and protocol-compatible Ledger tuple; Core beta.2 at `ce2324887f5b5ec4e3c2ec934ac874656d4c5348` requires Discovery protocol 20 and matching reflected descriptors before every Apply; its synced protobuf source matches Ledger beta.10. Qualify the selected beta.10 image, credentials and data compatibility on the exact Stack; this preparation does not authorize a Ledger upgrade. -4. Exercise existing-credential migration against an actual Ledger. Preserve key ID, Secret identity and namespace isolation. Verify the applied five-scope grant, continued source ingestion, durable cursor progress and denial of unrelated privileges. Credentials Ready plus observedGeneration proves distribution/spec observation; it does not acknowledge applied Ledger grants. -5. Validate the stack Auth issuer and authenticated API access, including allowed and denied tokens. During Auth transitions, route exposure must wait for the delegated Deployment/Service rollout proof. Without a Stack Auth module the existing contract is unauthenticated; explicitly select the intended sandbox configuration. - -If Connectivity remains in `LedgerCredentialsPending`, inspect the served Credentials CRD and Ledger operator revision first. A legacy CRD can prune `superuser`; the Operator may then repeatedly report an update and remain pending while legacy god grants remain active. This failure signature is inferred from source and must be exercised or rejected by runtime evidence; do not treat an old Ready status as success. - -Rolling back to v3.16.0 restores the previous superuser grants on reconciliation and the old Ledger API behavior. Treat this as privilege widening with a version-skew risk, requiring explicit operational approval and an exercised recovery plan that states the Ledger operator version and the LedgerConfiguration/Cluster schema used before and after recovery. Never infer safe rollback from Helm readiness. - -## Exact removed schema paths and OVH evidence limits - -Comparing the v1beta1 structural schema in `config/crd/bases/formance.com_ledgerconfigurations.yaml` at v3.16.0 and #548 (`0cdf98803609dd1510ce02bd79e44e23d209378d`) yields the following 35 removed property paths. `[]` marks array items; parent and child paths are listed separately. This inventory describes source schema changes, not effective live configuration. - -```text -spec.cluster.coldStorage -spec.cluster.coldStorage.bucketId -spec.cluster.coldStorage.driver -spec.cluster.coldStorage.path -spec.cluster.coldStorage.s3 -spec.cluster.coldStorage.s3.bucket -spec.cluster.coldStorage.s3.endpoint -spec.cluster.coldStorage.s3.region -spec.cluster.dnsEndpoint -spec.cluster.dnsEndpoint.annotations -spec.cluster.dnsEndpoint.enabled -spec.cluster.dnsEndpoint.endpoints -spec.cluster.dnsEndpoint.endpoints[].dnsName -spec.cluster.dnsEndpoint.endpoints[].providerSpecific -spec.cluster.dnsEndpoint.endpoints[].providerSpecific[].name -spec.cluster.dnsEndpoint.endpoints[].providerSpecific[].value -spec.cluster.dnsEndpoint.endpoints[].recordTTL -spec.cluster.dnsEndpoint.endpoints[].recordType -spec.cluster.dnsEndpoint.endpoints[].targets -spec.cluster.monitoring.pyroscope.authToken -spec.cluster.monitoring.pyroscope.basicAuthPassword -spec.cluster.monitoring.traces.sampling -spec.cluster.monitoring.traces.sampling.enabled -spec.cluster.monitoring.traces.sampling.successRatio -spec.cluster.persistence.coldCache -spec.cluster.persistence.coldCache.accessMode -spec.cluster.persistence.coldCache.hostPath -spec.cluster.persistence.coldCache.hostPath.path -spec.cluster.persistence.coldCache.hostPath.type -spec.cluster.persistence.coldCache.size -spec.cluster.persistence.coldCache.storageClass -spec.cluster.persistence.coldCache.volumeAttributesClassName -spec.cluster.receiptSigning -spec.cluster.receiptSigning.secretKey -spec.cluster.receiptSigning.secretName -``` - -`spec.cluster.monitoring.traces` remains present; only its `sampling` subtree is removed. `spec.cluster.persistence.data.accessMode` retains the same type and `ReadWriteOnce` default. Their apparent removal in a textual diff must not be treated as a removed schema path. The `spec.cluster.clusterID` property remains present but loses its `default: default` value; path comparison alone does not detect that changed default. - -The Infra owner's read-only OVH checkpoint reports zero explicit LedgerConfiguration objects. This does not prove that Settings, private Helm values, chart defaults, future manifests, or existing Ledger Cluster specs do not use removed configuration. Regions consumes a Secret through valuesFrom; no secret content was read or decrypted. Owners must compare authorized redacted effective values and Cluster specs with this schema before approving migration. - -The OVH Flux CI includes a shared `fluxcd.yml` template; its effective validation contract has not been inspected in this lane. No passing render/schema gate is inferred from that include. The Flux writer/grant, exact Stack, compatible published tuple, effective-values render, independent reviews and exercised runtime/recovery remain gates. No Flux edit, branch, push, manual deployment, AWS diff or cluster mutation is performed by this inventory. - -## Pyroscope credential migration before schema adoption - -The new schema replaces inline profiling credentials with Secret references. Each reference requires a non-empty Secret `name` and `key` and resolves in the delegated Ledger Cluster namespace: - -| Removed field under `spec.cluster.monitoring.pyroscope` | Replacement | -| --- | --- | -| `authToken` | `authTokenFrom.name` and `authTokenFrom.key` | -| `basicAuthPassword` | `basicAuthPasswordFrom.name` and `basicAuthPasswordFrom.key` | - -Keep `basicAuthUser` where basic authentication is used; that field remains supported. Retain the existing authentication method rather than enabling both modes as part of this migration. - -1. The configuration owner inventories which profiling authentication mode is used in the approved effective configuration, including Settings, private Helm values and existing Cluster specs. Record only field/reference presence and namespace ownership; never print credential values or copy them into this repository or review comments. -2. Before adopting the new schema, the owning GitOps lane prepares the existing credential material through its approved Secret-management mechanism in each relevant Cluster namespace. Do not remove the old configuration until the compatible Ledger operator can resolve the new references. This preparation document authorizes no Secret creation, decryption or cluster mutation. -3. Prepare the Secret selectors and the matching compatible operator/CRD rollout together. Do not apply the selectors to an old schema that prunes them, or drop the inline fields before the references can be used. Review the effective configuration without secrets and establish the explicit ordering/recovery plan before any separately authorized GitOps integration or reconciliation. -4. During separately authorized qualification, verify that profiling delivery still succeeds and that a missing Secret/key produces an actionable failure. Check that reference-only configuration survives admission and reconciliation without silently disabling profiling. Readiness alone does not establish successful authentication to the profiling service. -5. Recover using the approved operator/schema/configuration tuple if reference resolution or delivery fails. Preserve the original credential material in the approved Secret-management system for that recovery window; do not reconstruct inline credentials from this document or assume a downgrade understands the new selectors. - -This is a required migration procedure for existing profiling users, not evidence that OVH uses these fields or that any credential has been moved. The actual configuration inventory, Secret owner, rollout ordering and exercised recovery remain gates. - -## Preparation branch and delivery evidence - -The preparation PR targets `main`. A main-based release includes changes beyond the earlier isolated two-fix candidate; record the final source SHA and complete delta against v3.16.0 before tagging. The earlier baseline branch is historical preparation evidence, not the integration target. No `build-images` or `deploy-staging` label is part of preparation. - -The target is OVH sandbox. Changes are prepared for the owning GitOps path only; no manual deployment to these clusters is authorized, and GitOps merge/reconciliation/sync require separate authorization. Its actual Stack, GitOps source and effective context remain with the infrastructure lane to verify; AWS hosting/production is excluded. - -Track the retained credential contract through EN-2490 and the sandbox qualification through EN-2227 / EN-2231. This PR prepares a candidate only. Required evidence remains: pre-commit and relevant tests on the final tree, exact-head CI, independent Principal Engineer/Product Engineer/SRE review, code-owner approval, explicitly linked approval of the compatibility impacts, release authorization, published artifact digests, aligned Regions locks, and runtime qualification. No merge, tag, release, publication, deployment or environment synchronization is authorized by this document. From 666f6921e60746508e721d300d7eeca60dc43918 Mon Sep 17 00:00:00 2001 From: David Ragot <35502263+Dav-14@users.noreply.github.com> Date: Thu, 8 Oct 2026 18:06:57 +0200 Subject: [PATCH 7/7] chore: prepare Operator v3.17.0 minor release --- helm/crds/Chart.yaml | 4 ++-- helm/operator/Chart.lock | 6 +++--- helm/operator/Chart.yaml | 4 ++-- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/helm/crds/Chart.yaml b/helm/crds/Chart.yaml index 1ff1befa..b7d3a539 100644 --- a/helm/crds/Chart.yaml +++ b/helm/crds/Chart.yaml @@ -13,9 +13,9 @@ type: application # This is the chart version. This version number should be incremented each time you make changes # to the chart and its templates, including the app version. # Versions are expected to follow Semantic Versioning (https://semver.org/) -version: "3.16.1" +version: "3.17.0" # This is the version number of the application being deployed. This version number should be # incremented each time you make changes to the application. Versions are not expected to # follow Semantic Versioning. They should reflect the version the application is using. # It is recommended to use it with quotes. -appVersion: "v3.16.1" +appVersion: "v3.17.0" diff --git a/helm/operator/Chart.lock b/helm/operator/Chart.lock index 1969359c..9d718a94 100644 --- a/helm/operator/Chart.lock +++ b/helm/operator/Chart.lock @@ -1,6 +1,6 @@ dependencies: - name: operator-crds repository: file://../crds - version: 3.16.1 -digest: sha256:006ccb315031d902806e1d8479171f1def6848cfa866a72768ff75192beafbc7 -generated: "2026-10-07T11:18:50.016649+02:00" + version: 3.17.0 +digest: sha256:be121e8cf94b9d0eb4ea801d88e3dd7a1b442faf66c50de28a44076bc3dd06cc +generated: "2026-10-08T18:06:46.372994+02:00" diff --git a/helm/operator/Chart.yaml b/helm/operator/Chart.yaml index 57cd086b..7d8930d6 100644 --- a/helm/operator/Chart.yaml +++ b/helm/operator/Chart.yaml @@ -13,12 +13,12 @@ type: application # This is the chart version. This version number should be incremented each time you make changes # to the chart and its templates, including the app version. # Versions are expected to follow Semantic Versioning (https://semver.org/) -version: "3.16.1" +version: "3.17.0" # This is the version number of the application being deployed. This version number should be # incremented each time you make changes to the application. Versions are not expected to # follow Semantic Versioning. They should reflect the version the application is using. # It is recommended to use it with quotes. -appVersion: "v3.16.1" +appVersion: "v3.17.0" dependencies: - name: operator-crds version: "3.X"