Conversation
|
Adding a production data point from our management cluster, because we just hit a second and more severe failure mode from the same On 2026-09-05 a Task wedged in the
The connection is idle during exactly one window — while GitHub enumerates and counts objects server-side, before the packfile starts streaming. Your 1.94 GiB archival-branch case and this one look like the same root cause at two different severities: the wide ref set stretches the quiet period, and whether you get "9m37s" or "hangs until something else kills it" depends on whether a middlebox expires the flow first. So this change is an availability fix as well as a performance one. We have deployed a local admission-policy mitigation (injecting Happy to rebase or add a regression test if that would help it move. It is currently |
|
Additional validation from a Kelos v0.52.0 deployment: replacing We ran four sequential clone-only probes on the same Kubernetes node, with the same Git image (
Failures were exit 128 with This supports the change in this PR and did not show a mesh-dependent failure. It does not establish that the network was idle specifically because of server-side packing. One sample per cell per matrix, and the branch was moving rather than commit-pinned, so the durations are not a precise before/after performance benchmark. The matrix used node-local Follow-up: after deploying the admission policy, a fresh StatefulSet with a 20 GiB gp3 EBS workspace also cloned successfully in 6 seconds with ambient enabled. The generated StatefulSet template and resulting Pod both contained |
Kelos combined --depth 1 with --no-single-branch, which fetches the tip of every branch during workspace initialization. A repository with an unrelated 1.94 GiB archival branch spent 9m37s in the git-clone init container before a review Session could start, and on larger repos the clone could stall long enough for the pod-ready timeout to reap the Task. Use explicit --single-branch --depth 1 for both Job-backed Task workspaces and persistent WorkerPool/Session workspaces. Configured workspace refs are still selected with --branch, full commit SHA checkouts keep their targeted fetch path, and Task/Session branches continue to be fetched explicitly by branch-setup.
2d7a434 to
f97a35c
Compare
|
Squashed to a single commit ( Relationship to #1738. That issue is triaged It does not fix #1738's primary symptom (Sessions stranded in CI. @gjkim42 you triaged #1738 — would you be up for a look here, or point me at whoever owns the workspace-clone path? |
|
@scottt732 |
What type of PR is this?
/kind bug
What this PR does / why we need it:
Kelos currently combines
--depth 1with--no-single-branch, which fetches the tip of every branch during workspace initialization. A repository with an unrelated 1.94 GiB archival branch therefore spent 9m37s in thegit-cloneinit container before a review Session could start.This changes Job and persistent WorkerPool/Session workspace clones to explicit
--single-branch --depth 1. Configured workspace refs remain selected with--branch, full commit SHA checkouts keep their targeted fetch path, and Task/Session branches continue to be fetched explicitly bybranch-setup.Which issue(s) this PR is related to:
N/A
Special notes for your reviewer:
Verification:
make verifypasses.make buildpasses.TestClaudeEntrypointUsesPersistentSessionConfig; the same failure reproduces on unmodified upstreammain.Does this PR introduce a user-facing change?
Summary by cubic
Shallow-clones only the configured workspace branch so workspace initialization no longer downloads every branch tip. A 1.94 GiB archival branch previously added over nine minutes to clone time, and large clones could stall long enough for the pod-ready timeout to reap the Task.
Bug Fixes
--branch; full commit-SHA checkouts and Task/Sessionbranch-setupfetches are unchanged.Written for commit f97a35c. Summary will update on new commits.