fix(controller): fail locally invalid job templates

[Bluu (Bluuok)]

Malformed YAML or Jinja in a queued rollout's Job template can remain QUEUING indefinitely when cluster initialization, Job listing, or the submission quota prevents local validation from being reached.

Validate all queued manifests before Kubernetes initialization and listing, then reuse each valid manifest during submission. Invalid templates are reported as FAILED through the store; failed status updates are retried on the next reconciliation cycle. Valid jobs retain transient-cluster retry and quota behavior, and RUNNING jobs continue to be observed without re-rendering their templates. Update the _create_job docstring to describe terminal validation failures.

Validation on head 2e8e88563718993b3d73dda3504881ea22c2defe, reusing the existing Python 3.12 virtual environments:

• Ubuntu 24.04 CPU: python -m pytest -q tests/controller -p no:cacheprovider --basetemp .pytest-final-linux - 47 passed.
• Native Windows: python -m pytest -q tests/controller/test_k8s_invalid_templates.py tests/controller/test_k8s_reconciler.py --basetemp .pytest-final-windows - 24 passed.
• Scoped Ruff lint/format, Pyright (0 errors), repository header check, applicable cached pre-commit hooks, and git diff --check passed.

The new gate regressions on the prior head produced 14 failures with 7 passing controls. Coverage now exercises _reconcile_once under unavailable API/listing, full quota, mixed invalid/valid/running batches, transient recovery, manifest reuse, and status-update retry. The exploratory Windows full-controller run had 9 failures in unchanged local-process tests because os.killpg is unavailable; the complete controller directory passed on Ubuntu.

Tests use the real reconciliation loop, YAML/Jinja manifest builder, and status-patching method with simulated store HTTP and Kubernetes access. No real cluster, network, model inference, or GPU training was run. These are actual local checks from this repair pass; whole-repository CI has not run and remains awaiting maintainer approval (action_required, 0 jobs).

AI assistance was used; the production and regression diffs were reviewed, including an independent read-only review.

[Copilot]

Copilot review overview

🟡 Changes recommended

Real reconciliation can still contact Kubernetes or hit rate limiting before validating malformed templates.

Review effort: Balanced
Findings: 1 · 1

Open (2)

• Validate queued manifests before Kubernetes and rate-limit gates · New
• Update _create_job docstring for FAILED validation behavior · New

What changed in this PR

Fails queued rollouts when local Kubernetes Job template rendering or validation fails.

Changes:

• Separates local manifest validation from Kubernetes submission errors.
• Adds regressions for malformed YAML/Jinja, invalid kinds, and transient cluster errors.

File	Description
agentlightning/​controller/​k8s_reconciler.py	Marks local manifest failures as terminal.
tests/​controller/​test_k8s_invalid_templates.py	Tests invalid templates and retryable cluster failures.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.