Skip to content

Shallow clone only the configured workspace branch - #1711

Open
scottt732 wants to merge 1 commit into
kelos-dev:mainfrom
scottt732:fix/single-branch-shallow-clone
Open

scottt732 wants to merge 1 commit into
kelos-dev:mainfrom
scottt732:fix/single-branch-shallow-clone

Conversation

@scottt732

@scottt732 scottt732 commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

What type of PR is this?

/kind bug

What this PR does / why we need it:

Kelos currently combines --depth 1 with --no-single-branch, which fetches the tip of every branch during workspace initialization. A repository with an unrelated 1.94 GiB archival branch therefore spent 9m37s in the git-clone init container before a review Session could start.

This changes Job and persistent WorkerPool/Session workspace clones to explicit --single-branch --depth 1. Configured workspace refs remain selected with --branch, full commit SHA checkouts keep their targeted fetch path, and Task/Session branches continue to be fetched explicitly by branch-setup.

Which issue(s) this PR is related to:

N/A

Special notes for your reviewer:

Verification:

  • Focused Job, WorkerPool, full-SHA, and branch-setup regression tests pass.
  • make verify passes.
  • Integration suites pass: 151/151 controller specs and 12/12 install specs.
  • make build passes.
  • The full unit suite retains one pre-existing local fixture failure in TestClaudeEntrypointUsesPersistentSessionConfig; the same failure reproduces on unmodified upstream main.

Does this PR introduce a user-facing change?

Workspace initialization now shallow-clones only the configured branch instead of fetching every branch tip.

Summary by cubic

Shallow-clones only the configured workspace branch so workspace initialization no longer downloads every branch tip. A 1.94 GiB archival branch previously added over nine minutes to clone time, and large clones could stall long enough for the pod-ready timeout to reap the Task.

Bug Fixes

  • Applies to both Job-backed Tasks and persistent WorkerPool/Session workspaces.
  • Configured refs still select via --branch; full commit-SHA checkouts and Task/Session branch-setup fetches are unchanged.
  • Unrelated remote-tracking branches are no longer created; agents can fetch another branch explicitly if needed.

Written for commit f97a35c. Summary will update on new commits.

Review in cubic

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 7 files

Re-trigger cubic

@scottt732

Copy link
Copy Markdown
Contributor Author

Adding a production data point from our management cluster, because we just hit a second and more severe failure mode from the same --no-single-branch behavior — not a slow clone, but one that never finishes.

On 2026-09-05 a Task wedged in the git-clone init container and never reached Running. The pod sat in Init:0/2 until our client-side 5-minute pod-ready timeout reaped the whole Task and failed the user's request. We caught the retry live:

  • TCP to 140.82.113.3:443 ESTABLISHED, Recv-Q/Send-Q both 0, no RST
  • git index-pack read zero bytes over 30+ seconds/proc/<pid>/io rchar frozen at 4659, /workspace/repo frozen at 31996 bytes

Send-Q 0 is the tell: the client had nothing outstanding, so it never retransmitted and never learned the return path was dead. That is a silent idle-flow expiry on the NAT egress path (AWS NAT Gateway drops idle flows at 350s without sending an RST). git sets no low-speed timeout by default, so it waits indefinitely.

The connection is idle during exactly one window — while GitHub enumerates and counts objects server-side, before the packfile starts streaming. --no-single-branch is what makes that window long: cloneArgs at internal/controller/workerpool_controller.go:739 asks for the tip of every branch, which is 41 refs on the repository involved.

Your 1.94 GiB archival-branch case and this one look like the same root cause at two different severities: the wide ref set stretches the quiet period, and whether you get "9m37s" or "hangs until something else kills it" depends on whether a middlebox expires the flow first. So this change is an availability fix as well as a performance one.

We have deployed a local admission-policy mitigation (injecting http.lowSpeedLimit/http.lowSpeedTime into the init container) to convert the hang into a fast, retryable failure — but that is a downstream workaround for the symptom, not the cause this PR addresses.

Happy to rebase or add a regression test if that would help it move. It is currently MERGEABLE with no conflicts, but has sat in needs-triage/needs-actor since Aug 31.

@scottt732

scottt732 commented Sep 9, 2026

Copy link
Copy Markdown
Contributor Author

Additional validation from a Kelos v0.52.0 deployment: replacing --no-single-branch with --single-branch resolved repeated clone startup failures for a large private repository.

We ran four sequential clone-only probes on the same Kubernetes node, with the same Git image (alpine/git:v2.47.2), credentials, repository/branch, depth 1, security context, and fresh disk-backed emptyDir per probe. An unchanged Git low-speed guard aborted transfers below 100 B/s for 120 seconds. We repeated the matrix after upgrading Istio to 1.31.0 and verified ambient enrollment/opt-out through the node-local ztunnel.

Clone scope Ambient mesh First run After Istio 1.31.0 upgrade
All branch tips Enrolled Failed, 127s Failed, 123s
Selected branch only Enrolled Passed, 9s Passed, 6s
All branch tips Opted out Failed, 123s Failed, 123s
Selected branch only Opted out Passed, 7s Passed, 6s

Failures were exit 128 with curl 28 Operation too slow, followed by early EOF / invalid index-pack output; successes exited 0. We used a narrowly scoped Kyverno admission mutation to replace only the exact clone flag, preserving the rest of the command.

This supports the change in this PR and did not show a mesh-dependent failure. It does not establish that the network was idle specifically because of server-side packing. One sample per cell per matrix, and the branch was moving rather than commit-pinned, so the durations are not a precise before/after performance benchmark. The matrix used node-local emptyDir; the original Session uses gp3 EBS. The original all-branch clone did eventually succeed on a later retry, so the problem is intermittent rather than universal.

Follow-up: after deploying the admission policy, a fresh StatefulSet with a 20 GiB gp3 EBS workspace also cloned successfully in 6 seconds with ambient enabled. The generated StatefulSet template and resulting Pod both contained --single-branch; the clone exited 0.

Kelos combined --depth 1 with --no-single-branch, which fetches the tip
of every branch during workspace initialization. A repository with an
unrelated 1.94 GiB archival branch spent 9m37s in the git-clone init
container before a review Session could start, and on larger repos the
clone could stall long enough for the pod-ready timeout to reap the
Task.

Use explicit --single-branch --depth 1 for both Job-backed Task
workspaces and persistent WorkerPool/Session workspaces. Configured
workspace refs are still selected with --branch, full commit SHA
checkouts keep their targeted fetch path, and Task/Session branches
continue to be fetched explicitly by branch-setup.
@scottt732

scottt732 commented Sep 17, 2026

Copy link
Copy Markdown
Contributor Author

Squashed to a single commit (f97a35c) and dropped two local planning artifacts under docs/superpowers/ that should not have been in the diff. The change is now 5 files, +41/-6 — two --no-single-branch--single-branch call sites plus regression coverage.

Relationship to #1738. That issue is triaged kind/bug / priority/important-soon and covers two symptoms in the workspace-startup path. This PR fixes the second one: the --no-single-branch --depth 1 clone stall I documented in this comment, where a Task wedged in the git-clone init container at Init:0/2 and never reached Running until the pod-ready timeout reaped it. Four sequential clone-only probes on the same node, image, credentials, repo, branch, and depth showed --single-branch resolving it.

It does not fix #1738's primary symptom (Sessions stranded in Pending/IdleResumeRequested with no pod) — that mechanism is still unconfirmed and needs separate work.

CI. fork-e2e / e2e-with-environment on this PR never actually ran: it sat queued for ~297h awaiting environment approval and was then cancelled in what looks like a sweep of stale runs on Sep 13. That leaves pr-e2e permanently pending, which is why the PR reads BLOCKED — it is not a test failure. A maintainer approving the fork workflow run would unstick it.

@gjkim42 you triaged #1738 — would you be up for a look here, or point me at whoever owns the workspace-clone path?

@gjkim42

gjkim42 commented Sep 19, 2026

Copy link
Copy Markdown
Collaborator

@scottt732
I rebase onto main branch several times during a task or a session. Is it compatible with that workflow?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants