Allow precise-code-intel-worker /tmp scratch on a per-pod PVC - #956
Merged
Merged
Conversation
devdinu
marked this pull request as ready for review
September 30, 2026 23:48
The worker writes large SCIP uploads to /tmp before processing them in multiple passes, and /tmp was always an unbounded emptyDir. That puts the write on the node boot disk, where a large upload competes with every other pod on the node and can fill the disk. A new storageType value selects the backing store. emptyDir stays the default, so rendered output is unchanged for existing installs. pvc switches /tmp to a generic ephemeral volume, giving each pod its own claim on storageClass.name that is discarded with the pod. storageSize is required for pvc and sets sizeLimit when left on emptyDir. The mount path stays /tmp, so the worker needs no TMPDIR redirect. A freshly provisioned volume is root-owned and the worker runs as a non-root user, so the pvc branch defaults the pod fsGroup to the container runAsGroup (with fsGroupChangePolicy OnRootMismatch) to keep /tmp writable. A user-set podSecurityContext.fsGroup still wins. An unknown storageType now fails rendering instead of silently falling back to emptyDir. Amp-Thread-ID: https://ampcode.com/threads/T-01a0f4b6-78bf-7109-82dc-4545a652ecdf Co-authored-by: Dinesh Kumar <dinesh.kumar@sourcegraph.com>
filiphaftek
approved these changes
Oct 1, 2026
filiphaftek
left a comment
filiphaftek
left a comment
Contributor
There was a problem hiding this comment.
Very nice!
I did some follow up in diff-your, and looks like using 100GB size we can increase 4x throuput + fully offload the boot disk: https://sourcegraph.sourcegraph.com/deepsearch/a9003fa4-3d60-408c-8931-57ddc3034cbc
devdinu
force-pushed
the
09-30/pci-worker-pvc-storage
branch
from
October 2, 2026 01:05
1230ddf to
bc7adc8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The precise-code-intel-worker writes large SCIP uploads to /tmp before processing them in multiple passes, and /tmp was always an unbounded emptyDir. That places the write on the node boot disk, where a large upload competes with other pods on the node and can fill the disk.
Changes
preciseCodeIntel.storageTypeto select the backing store for the/tmpscratch volume. One ofemptyDir(default) orpvcpvcmounts a generic ephemeral volume, so each pod gets its own claim onstorageClass.namethat is created and deleted with the podpreciseCodeIntel.storageSize, required forpvcand applied assizeLimitwhen left onemptyDirThe default renders
emptyDir: {}exactly as before, so existing installs are unaffected. The mount path stays/tmp, so the worker needs no TMPDIR redirect.ref https://app.incident.io/sourcegraph/response/incidents/531
ref EPD2-427
Test Plan
Will follow up with controller PR to enable this based on toggle.