sec(#947): chart-manage the mysql-client root password (rotate off the baked literal)

[saqlainsyed007]

What

backend#947 / backend#1528 Phase 0. Enable rotating the mysql-client root password off the image's baked literal. A new rotateMysqlRoot gate (default off, ByEnv-resolved exactly like serviceDbAccounts / bootstrapDbReparent): when on, the chart generates a random root password into the Secret (3-tier: pin → existing Secret → randAlphaNum, mirroring credmgr) and injects it as MYSQL_ROOT_PASSWORD on the mysql-client deployment.

This is the credential the #785 re-parent points the mint at, and — separately — a literal now disclosed in public git history, so rotating it is both a Phase-0 prerequisite and a standing-exposure fix.

Scope — read this, it's the important part

This rotates a FRESH datadir only. The mysql entrypoint reads MYSQL_ROOT_PASSWORD at first init and ignores it thereafter, so an existing edge keeps its current root password until the one-time rollout DDL runs:

ALTER USER 'root'@'%'        IDENTIFIED BY '<the generated Secret value>';
ALTER USER 'root'@'localhost' IDENTIFIED BY '<the generated Secret value>';

(read the value from the Secret, run via kubectl exec as root@localhost). The chart deliberately does not run that ALTER — it would have to authenticate as root with the current literal (re-introducing it) or hit a chicken/egg once rotated. Consistent with the rest of #1528, the live-fleet DDL stays an ops step; the chart only provides the Secret + the fresh-init override.

Turning the gate on rolls the mysql pod (an env change → brief restart), so flip it in a window. And any tooling/runbook that dials root with the old literal must move to the Secret first, or it breaks at rotation — that inventory is the substance of #947.

Changes

• values.yaml — rotateMysqlRoot (override) + rotateMysqlRootByEnv{dev,stg,prod: false} + mysqlRootPassword pin, with the fresh-vs-existing distinction and the ALTER step documented inline.
• _helpers.tpl — tracebloc.rotateMysqlRoot, resolving like serviceDbAccounts.
• secrets.yaml — MYSQL_ROOT_PASSWORD 3-tier generate-and-persist, gated. No alphanumeric constraint (connection param, never DDL); placeholder-guarded.
• mysql-deployment.yaml — MYSQL_ROOT_PASSWORD env from the Secret, gated. Probes use mysqladmin ping (needs no auth), so they're undisturbed.
• values.schema.json — three keys. Chart.yaml 1.9.68 → 1.9.69 (+ appVersion lockstep).
• Tests — +7 helm-unittest cases.

Design notes

• Generate-and-persist, not require-pin — a strong random password with no literal anywhere, matching every sibling secret. (The require-pin shape feat(#1528): wire DB_BOOTSTRAP_* into jobs-manager to re-parent the mint off edgeuser (S3) #785 uses is right there because jobs-manager authenticates as root and can't ALTER it; here the chart owns the value, so generate is safe and better.)
• Flag-gated, default-off — mirrors every #1528 flag; default installs stay byte-identical, and flipping it (which rolls mysql) is a deliberate windowed action.
• bootstrapDbPassword wiring is a follow-up — kept out of scope so this is one self-contained change and feat(#1528): wire DB_BOOTSTRAP_* into jobs-manager to re-parent the mint off edgeuser (S3) #785 (merged hours ago) isn't churned. During rollout the operator points bootstrapDbPassword at this Secret's value; a later PR can unify them.

Verification

• helm unittest full suite: 569 passed (34 suites).
• helm template (ci/eks base): default render = 0 MYSQL_ROOT_PASSWORD refs (byte-identical); gate-on injects a 48-char random password (decoded, confirmed not the literal) into the Secret and the deployment env; rotateMysqlRootByEnv.dev=true resolves on.
• helm lint clean; version-bump guard satisfied; no Edg9@Tr@ce in the diff.

Not in this PR (the rest of Phase 0 / #947)

• The per-fleet ALTER USER 'root' rollout (ops, reads this Secret).
• The root-consumer inventory (runbooks, DR/backup tooling) — must move off the literal before rotating.
• The Dockerfile.mysql_client ENV source scrub (gated on the client#454 image freeze; the runtime override here neutralizes the baked default for new installs meanwhile).
• The client-runtime Service Bus key (separate #947 scope).

Refs: backend#947 · backend#1528 (Phase 0) · client#785 (the re-parent this feeds).

🤖 Generated with Claude Code

---

Note

Medium Risk
When enabled, mis-rotation or flipping the gate without the runbook can lock out root consumers or briefly roll mysql; default-off limits blast radius but fleet rollout is credential-sensitive.

Overview
backend#947 / backend#1528 Phase 0 adds an opt-in path to stop relying on the mysql-client image’s baked-in root password. A new rotateMysqlRoot gate (default off, resolved like serviceDbAccounts via rotateMysqlRootByEnv) controls whether the chart owns root credentials.

When the gate is on, the release Secret gets MYSQL_ROOT_PASSWORD (pin → existing Secret → randAlphaNum), and the mysql-client Deployment reads it as MYSQL_ROOT_PASSWORD. That only applies on a fresh datadir; live edges still need a one-time ops ALTER USER 'root' using the Secret (documented in docs/migration-tools/rotate-mysql-root.md). Default installs stay byte-identical (no env block, no Secret key).

Also documents MYSQL_ROOT_PW for migrate-tenant.sh when root is chart-managed, extends values.yaml / values.schema.json, bumps the chart to 1.9.71, and adds helm-unittest coverage for the gate and password validation.

^{Reviewed by Cursor Bugbot for commit 3693c57. Bugbot is set up for automated code reviews on this repo. Configure here.}

[saadqbal]

Approving — and the scope statement is the best part of this. Saying plainly that the chart rotates a fresh datadir only, that it deliberately does not run the ALTER USER, and why (authenticating as root with the current literal would re-introduce it, or hit a chicken/egg once rotated) is worth more than a fix that quietly implied the exposure was closed. Same for flagging that the gate flip rolls the pod, and that any tooling dialling root with the old literal breaks at rotation.

Checked the things that would hurt on a public repo or a live edge:

Generate-once is real. $existingSecret at secrets.yaml:17 is a genuine lookup, so tier 2 preserves the value across upgrades — no randAlphaNum-on-every-upgrade trap locking MySQL out of its own datadir.

The Secret key and the secretKeyRef are gated by the same helper, so they appear and disappear together. A default install can't end up with a deployment referencing a key the Secret didn't emit.

Nothing shipped. values.yaml carries mysqlRootPassword: "", not a default, and the only literals in the diff are helm-unittest fixtures in tests/secrets_test.yaml. The placeholder guard on the pin is a nice touch — <ROOT_PASSWORD> failing loudly beats it reaching MySQL.

--reuse-values lands safe. rotateMysqlRoot is unset-vs-false aware via kindIs "invalid", and both it and rotateMysqlRootByEnv are default dict-guarded — so an upgrade that carries stored values without the new keys resolves to gate-off rather than crashing templating or half-rotating.

To be precise about what this does and doesn't close: it's the enabling half. The literal is still in public history and every existing edge still has it live until the rollout DDL runs, which is exactly how the body frames it.

[saqlainsyed007]

bugbot run

[saqlainsyed007]

bugbot run

[saqlainsyed007]

bugbot run

[saqlainsyed007]

bugbot run

[saqlainsyed007]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 3693c57. Configure here.}

[saadqbal]

Re-approving. My earlier approval was dismissed by the pushes since; I've read all five.

The alphanumeric constraint is the one I looked at hardest, because narrowing a password charset is normally a step backwards. Here it isn't: the value lands inside ALTER USER … IDENTIFIED BY '…' in the rotation runbook, so a quote or shell metacharacter breaks the one-time rotation — and it breaks it mid-way, on the operation you least want half-applied. Rejecting it at template time beats discovering it in a DDL statement.

What makes it right rather than merely defensible is the schema note drawing the line explicitly: constrained here, not constrained for bootstrapDbPassword, because that one is only ever a connection parameter and this one also lands in DDL. Correcting the old "NOT constrained to alphanumeric" sentence in the same change matters too — that claim was about to become false, and it's precisely the sentence a future operator would have relied on. randAlphaNum already satisfying the constraint means the generated path is unaffected, so this only ever fires on an operator pin.

The three runbook commits are the same instinct applied to the procedure: failing loudly on a failed ALTER, asserting 1045 on the old-password check rather than assuming the negative, and refusing an empty root password. A rotation runbook that continues past a failed step is worse than one that stops.

20 checks green, no open threads.