fix(chart): point jobs-manager readiness at /healthz, now that it can go red (backend#1779)

[LukasWodka]

Summary

Repoints jobs-manager's readiness probe from tcpSocket: 8080 to httpGet: /healthz on 8080, now that /healthz means something. This is step 2 of backend#1779; step 1 (tracebloc/client-runtime#327, merged 2026-08-14) is what makes it safe, and step 3 (assert the probe can go red) is the test half below.

The ordering matters and it is the thing this ticket warned about. When client#699 added the probe on 2026-08-13, /healthz was still return jsonify({"status": "ok"}), 200 — probing it then would have swapped "no probe, state obviously unknown" for "a probe says healthy", which is the backend#1729 class wearing the uniform of a fix. The TCP connect was the correct call at the time. It is no longer the best available signal.

Related

Fixes tracebloc/backend#1779 (step 2 + step 3; the pods-monitor / resource-monitor / egress-proxy decision from "Also worth deciding" is still open, and the pods-monitor half is recorded as a chart test rather than left as an omission).

Depends on tracebloc/client-runtime#327 — already merged, so there is no rollout ordering to manage here.

What /healthz actually returns (verified, not assumed)

Read from origin/develop:submit_ingestion_run.py (_healthz_report, line 1694) and origin/develop:sql_utils.py in client-runtime:

condition	status
authz_mount, schema and metadata_db all "ok"	200 {"status":"ok","checks":{…}}
any one of them not "ok"	503 {"status":"unavailable","failed":[…],"checks":{…}}

• authz_mount — Path(ctx.authz_path).is_file(). A missing mount means load_authz_policy fell back to its deny-all empty policy and every caller gets 403 forever. An empty policy (allowed: []) is deliberately still ok — that is this chart's fail-safe render for an operator who has not configured ingestion yet, so a fresh edge stays legitimately Ready.
• schema — the vendored ingest schema validator is loaded.
• metadata_db — ping_metadata_db() runs SELECT 1.

No before_request hook and no auth on the route, so an unauthenticated kubelet GET works. Kubernetes treats 200–399 as success and anything else as failure, so 503 is exactly the "not ready" signal the probe needs. 200/503 is the contract this probe keys on. All three failures happen with 8080 bound and answering, which is precisely what the TCP connect could not see.

timeoutSeconds goes 3 → 5, derived from the endpoint rather than picked: ping_metadata_db passes connection_timeout=2 and read_timeout=2, so a half-open MySQL takes ~4s to raise. Its docstring says the check must finish inside the probe's budget, because "a health check that outlives its own probe timeout reports 'timeout', not 'database down', and the two want different operator responses." Both outcomes mark the pod NotReady; only one of them says why.

READINESS ONLY — unchanged, and now better supported

No livenessProbe and no startupProbe, and the chart comment explaining that is preserved. /healthz makes the argument stronger, not weaker, and _healthz_report's own docstring says so: its metadata_db check makes the answer depend on MySQL, which is "safe for READINESS and wrong for LIVENESS or a startupProbe". On either of those a MySQL blip would restart jobs-manager and take Service Bus polling and training submission down with it, reversing the decision jobs_manager.py makes on purpose when it catches a failed run_server_in_thread. Two "must not exist" tests guard that, and two of the mutations below prove they bite.

The INGESTION_HTTP_DISABLED gate is preserved verbatim, with the same truthiness on both sides (not os.getenv(...) in the runtime, non-empty-string-is-truthy in Helm), and a mutation covers removing it.

The backend#1964 interaction — thought about, not ignored

Yes, this widens the set of states in which a previously-Ready pod is now NotReady, and yes, that can freeze image refresh sooner. I think it is the right trade, with one thing worth knowing.

image-refresh-cronjob.yaml skips a tick while kubectl rollout status says the jobs-manager Deployment is unsettled, counts the skips in tracebloc.io/refresh-skip-streak, and at MAX_SKIP_TICKS (8, ~2h at the 15m schedule) fails the tick with a loud ERROR instead of exiting 0 forever.

Newly-NotReady states, and what I concluded about each:

1. authz ConfigMap not mounted and no schema validator — these are already broken pods: every caller gets 403 or the app is half-constructed. Pausing image refresh on them is correct, and #1964's ceiling is what turns the pause into a page. The 2026-08-11 incident is the argument for making these visible, not for keeping them silently Ready.

2. MySQL unreachable — this is the genuinely new one. Previously a jobs-manager whose MySQL went down after startup stayed Ready and image refresh kept running. Now it goes NotReady after failureThreshold: 3 × periodSeconds: 10 = 30s of consecutive failures, and if MySQL stays down past ~2h the skip streak trips and image refresh fails the tick for all four workloads.

   I judged that acceptable and, on balance, correct. A two-hour MySQL outage on an edge is already a full outage — jobs-manager cannot record an ingestion run, so it would answer 500s to every submission anyway, and re-imaging control-plane pods during it is not something we want to be doing quietly. #1964's ceiling exists exactly so a long freeze becomes an ERROR someone reads rather than a green CronJob nobody looks at, so the new path lands in the loud branch, not the silent one. The remediation is to fix MySQL, not to raise MAX_SKIP_TICKS.

3. Not new: failureThreshold: 3 is unchanged, so a single slow probe still does not flip the pod. And nothing here kills the container — a NotReady jobs-manager keeps polling Service Bus and keeps training, which is the whole point of the readiness-only decision.

MAX_SKIP_TICKS is left at 8. Raising it to buy headroom for MySQL outages would trade a loud finding for a longer silence, which is the failure mode #1964 was filed against.

Because that comment block asserted "jobs-manager's readiness probe … is a TCP connect to 8080" and enumerated its never-settles triggers from that premise, this PR updates it in the same commit — the house rule is that a change which falsifies a statement fixes it in place. The frozen-refresh remediation log gets the same treatment: it now tells the operator the probe is /healthz, that 8080 may be open with the probe still red, and that the 503 body names which dependency failed.

On step 3, "assert the probe can go red" — where the halves live

Being precise, because "we added tests" is the easy way to overclaim here. Chart tests can only assert the probe's shape; they cannot make a kubelet observe a 503. The two halves:

• The endpoint can go red — already covered in client-runtime by Sync develop → main for v1.9.2 chart release (installer cred-leak fix + curl|bash survival) #327: test_healthz_503_when_metadata_db_is_unreachable, test_healthz_503_when_the_authz_configmap_is_not_mounted, test_healthz_reports_every_failing_check_not_just_the_first, plus the negative cases (test_healthz_stays_ready_with_an_empty_authz_policy, ..._with_no_ingestor_image_configured) that pin what must not turn it red.
• The probe is wired at that endpoint — this PR, mutation-proven below.

What no test here covers is the kubelet actually derouting the pod on a 503; that is Kubernetes' own contract (non-2xx/3xx = failure) and would need a live cluster. backend#1311 (FR-assist per staging hop) is the place that would land if we want it observed end to end.

Test plan

client/tests/jobs_manager_test.yaml, 542 → 544 chart tests. Two new cases plus rewritten assertions on the existing ones:

• should give the api container a readiness probe on GET /healthz — path and port asserted separately so a failure says which half broke.
• should NOT fall back to a bare TCP connect on 8080 (new) — kept as its own named case. It is honestly not catching a revert the httpGet assertions miss (they redden too, and Kubernetes rejects a probe carrying two handler keys outright); it is there so one of the failures reads tcpSocket expected to NOT exists rather than three unknown path …httpGet.* lines that look like a typo. A revert to the weaker signal should not need diagnosing. It is also the only assertion here independent of what replaced the httpGet block.
• should let /healthz outlive a half-open MySQL before the probe times out (new) — pins timeoutSeconds: 5, derived from connection_timeout=2 + read_timeout=2 in ping_metadata_db, not chosen.
• should NOT give the api container any probe that restarts it (deliberate) — livenessProbe and startupProbe must not exist.
• Gate tests both ways (INGESTION_HTTP_DISABLED: "1" drops the probe, "" keeps it) and the pods-monitor recorded-decision test.

$ helm unittest ./client
Charts:      1 passed, 1 total
Test Suites: 34 passed, 34 total
Tests:       544 passed, 544 total

Baseline on develop (same command, changes stashed): 542 passed, 542 total.

Also green, unchanged: make lint (51 scripts parse, shellcheck 58 files), make drift (all 14 guards), make helm-lint, make helm-vocab (28 checks), make helm-template (aks/bm/eks/oc), and the four bats files that render the chart — chart-pull-secret, image-refresh-skip-streak, hostpath-prep, pipefail-early-close — 81 tests, 0 failures. (The full 1325-test bats suite is ~20 min; a partial local run reached 626 ok / 0 not ok before I cut it. CI runs it in full.)

Mutation proof

Seven mutations against jobs-manager-deployment.yaml. The harness asserts each anchor matched exactly once before running the suite — an inert mutation and real coverage produce an identical green log, so a stale anchor is reported as a failure rather than as a pass.

#	mutation	result
1	revert to tcpSocket: 8080	RED — 3 failed, 51 passed
2	path: /healthz → path: /	RED — 2 failed, 52 passed
3	port: 8080 → port: 9090	RED — 2 failed, 52 passed
4	add a startupProbe (the client#699 bug)	RED — 1 failed, 53 passed
5	add a livenessProbe	RED — 1 failed, 53 passed
6	timeoutSeconds back to 3	RED — 1 failed, 53 passed
7	drop the INGESTION_HTTP_DISABLED gate	RED — 1 failed, 53 passed
—	restored	GREEN — 54 passed, 54 total

Mutation 1 in full, since it is the regression this ticket exists to prevent:

MUTATION APPLIED: tcpsocket
 FAIL  Jobs Manager Deployment	client/tests/jobs_manager_test.yaml
	- should give the api container a readiness probe on GET /healthz
		- asserts[1] `equal` fail
			Error: unknown path spec.template.spec.containers[0].readinessProbe.httpGet.path
		- asserts[2] `equal` fail
			Error: unknown path spec.template.spec.containers[0].readinessProbe.httpGet.port
	- should NOT fall back to a bare TCP connect on 8080
		- asserts[0] `notExists` fail
			Path: spec.template.spec.containers[0].readinessProbe.tcpSocket expected to NOT exists
	- should still probe when INGESTION_HTTP_DISABLED is present but empty
		- asserts[0] `equal` fail
			Error: unknown path spec.template.spec.containers[0].readinessProbe.httpGet.path
		- asserts[1] `equal` fail
			Error: unknown path spec.template.spec.containers[0].readinessProbe.httpGet.port
Tests:       3 failed, 51 passed, 54 total

Rendered output, helm template t ./client -f client/ci/eks-values.yaml:

        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080

Type of change

• Bug fix

Deployment notes

client/Chart.yaml bumped 1.9.63 → 1.9.64 (chart-version guard: release-helm-chart.yaml packages ./client into the shared index.yaml, and a Helm repo only ever publishes a new version).

Otherwise chart-only; no env vars, no migrations, no flags. On upgrade the jobs-manager pod rolls and the new probe applies immediately. The image it runs must include client-runtime#327 (merged 2026-08-14) — an older image serves the unconditional 200 and the probe simply behaves as it did before, so there is no ordering hazard, only a loss of the new signal.

Operators on an edge with a genuinely broken dependency (unmounted ingestion-authz ConfigMap, unreachable MySQL) will see jobs-manager report NotReady where it previously reported Ready. That is the intended change, and kubectl describe pod plus the 503 body name which check failed.

Checklist

• Tests added / updated and passing locally
• Docs updated if behavior or config changed — the backend#1964 comment block and the frozen-refresh remediation log in image-refresh-cronjob.yaml, both of which this change falsified
• No secrets / credentials in the diff
• Cross-repo issues use Fixes tracebloc/<repo>#N
• Dependency already shipped (client-runtime#327 merged), so no expand-then-contract sequencing needed
• Terminal output follows STYLE.md — bash scripts/check-style.sh passes (via make lint)

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes jobs-manager readiness, so previously Ready pods can go NotReady on MySQL/authz failures and pause image refresh until they settle. Still readiness-only (no liveness/startup), so training and Service Bus polling are not killed.

Overview
Repoints the jobs-manager readiness probe from tcpSocket: 8080 to httpGet: /healthz (timeout 3s → 5s), now that /healthz returns 503 when authz_mount, schema, or metadata_db fail. A bound port is no longer treated as ready to serve POST /internal/submit-ingestion-run.

Still readiness only — no liveness or startup probe — so a failed ingestion HTTP start does not CrashLoop the pod or stop Service Bus polling/training. The INGESTION_HTTP_DISABLED gate is unchanged.

Image-refresh skip-streak comments and operator logs now describe /healthz (including curling the 503 body) and note that a long MySQL outage can freeze refresh after MAX_SKIP_TICKS. Chart tests pin httpGet path/port, timeoutSeconds: 5, and tcpSocket must not exist. Chart version 1.9.63 → 1.9.64.

^{Reviewed by Cursor Bugbot for commit cbe25b4. Bugbot is set up for automated code reviews on this repo. Configure here.}

[saadqbal]

The probe change itself is the right fix and well argued — /healthz instead of tcpSocket is what stops the Service routing POST /internal/submit-ingestion-run at a pod whose server isn't listening, and the note about readiness failing being harmless (nothing kills a NotReady pod, so a slow migration just delays Ready) is the important half, because that's what makes a longer probe safe here.

Not approving yet, though, because Bugbot's two Mediums on the remediation block are both correct, and I checked rather than took them:

The label selector matches nothing. The pod template in jobs-manager-deployment.yaml sets exactly app: manager (line 32), and tracebloc.labels — which is applied to the Deployment's own metadata, not the pod template — emits app.kubernetes.io/{name,instance,version,managed-by} plus helm.sh/chart and no component at all. So all three -l app.kubernetes.io/component=jobs-manager commands in that block return nothing, including the describe pod ... | grep -A2 'Readiness probe failed' line that exists specifically to surface the failure. -l app=manager is the selector that works.

And the 503-body advice can't be followed as written. Kubelet probe-failure events carry the status code, not the response body — you get Readiness probe failed: HTTP probe failed with statuscode: 503 and nothing else. So describe pod will never show authz_mount / schema / metadata_db, which is the one piece of information that line is telling the operator to go and read. If /healthz names the failed check in its body, the way to see it is from inside the pod, e.g.

kubectl -n <ns> exec deployment/<name> -c api -- curl -sS -o- -w '\n%{http_code}\n' localhost:8080/healthz

or the container's own log line for the failed check, if it emits one.

Worth fixing properly rather than trimming the advice: this block only ever runs when refresh has been frozen for MAX_SKIP_TICKS, so it's read by someone already in an incident, and a command that silently returns nothing is worse than no command — it reads as "the pod is fine".

Nothing else in the diff concerns me. Re-request me once those two are addressed and the threads are closed out.

[LukasWodka]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit cbe25b4. Configure here.}

[saqlainsyed007]

Verified the probe change, the ordering rationale, and the test — a correct readiness fix with the sequencing right.

Repointing jobs-manager's readiness from tcpSocket: 8080 to httpGet: /healthz is only safe now: the probe shipped as a TCP connect (client#699) precisely because /healthz was then an unconditional 200 {"status":"ok"} — pointing a probe at it would have swapped "no probe, obviously unknown" for "a probe says healthy" that could never go red, the backend#1729 class in disguise. client-runtime#327 made /healthz report authz_mount/schema/metadata_db and 503 on failure, so it's now the stronger signal, and this is the correct follow-up. The chart ships readiness ONLY (no liveness/startupProbe, which would kill the deliberately-catch-a-failed-server pod).

The test asserts the specific thing both ways: a case pinning readinessProbe.httpGet.path = /healthz (+ port) and a separate "should NOT fall back to a bare TCP connect on 8080" — so a regression to tcpSocket reddens rather than passing. The image-refresh never-settles ceiling comment is correctly widened: a pod with an unmounted ingestion-authz ConfigMap or unreachable MySQL is now NotReady where the TCP probe answered Ready, and the ceiling turns a long outage into a loud ERROR (fix the named dependency, don't raise MAX_SKIP_TICKS). Operator diagnostics now explain that the kubelet event carries only the status code, so port-forward + curl /healthz names the failed dependency. Chart version bumped, MIGRATION.md updated. CI green, no threads. LGTM.