Skip to content

Kubernetes engines pull cached models through the mirror when egress is off - #410

Merged
Priyanshu-u07 merged 1 commit into
mainfrom
fix/k8s-pods-use-model-mirror
Oct 14, 2026
Merged

Priyanshu-u07 merged 1 commit into
mainfrom
fix/k8s-pods-use-model-mirror

Conversation

@Priyanshu-u07

Copy link
Copy Markdown
Collaborator

With model egress off, a Kubernetes deploy passed the cache gate but the engine pod was never pointed at the mirror. Only the worker path applied the mirror step. The direct path built the pod from the stored configuration alone, so on an air-gapped cluster the pod would try huggingface.co and never become ready.

The direct path now runs the same mirror step before creating the pod, so a cached model gets HF_ENDPOINT in the pod's environment.

  • Only when model egress is off. Connected installs keep pulling from origin, since their mirror base may not be reachable from inside the cluster.
  • Kubernetes only. Nosana and Akash nodes are on the public internet and cannot reach the control plane.
  • With egress off, INFERIA_MODEL_MIRROR_BASE must be an address the pods can reach.
  • Ollama on Kubernetes is unchanged for now. Its start script runs its own pull and needs a separate change.

Tests cover the Kubernetes case, a connected install, the public providers, no mirror base, an uncached model, and Ollama.

Related: #311

Signed-off-by: Priyanshu-u07 <connect.priyanshu8271@gmail.com>
@Priyanshu-u07
Priyanshu-u07 merged commit 51f8d9c into main Oct 14, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant