Skip to content

Feat | Limit Kustomize streams to declared dependencies - #514

Open
ak-a wants to merge 5 commits into
dag-andersen:mainfrom
ak-a:perf/stream-kustomize-dependencies
Open

ak-a wants to merge 5 commits into
dag-andersen:mainfrom
ak-a:perf/stream-kustomize-dependencies

Conversation

@ak-a

@ak-a ak-a commented Nov 2, 2026 •

Copy link
Copy Markdown

Kustomize sources currently stream the full checkout for each render. The archive cache avoids repeated compression, but the repo server still receives and extracts unrelated files.

Use argocd.argoproj.io/manifest-generate-paths to stream the source directory and its declared dependencies. Preserve the repository layout and both literal and glob matches. Cache archives by their final exclusions, so full-checkout fallbacks share one archive. Write scoped archives with literal path lookups and skip excluded directories before reading metadata.

Narrowing requires an explicit Kustomize source (kustomize: {} is sufficient). Sources that use automatic discovery, lack the annotation, or retain symlinks keep the full checkout. Source override files also disable narrowing. Annotation whitespace is normalized as in application selection. All required shared bases, patches, and value files must be declared.

In one comparison of the initial implementation, a monorepo with about 18,000 files rendered 24 applications at two revisions, with concurrency 20 and an existing Argo CD instance:

Measurement v0.2.16 Candidate
Preview action 71s 15s
Manifest rendering, 48 renders 59s 2s
Files per streamed archive 17,945 21

All 48 rendered manifest files were byte-for-byte identical. Both runs used the same input commits; the candidate used main plus this change. The baseline ran first and backend caches were not reset. The observed 79% reduction applies to this workload and predates the review fixes.

A local synthetic benchmark of the final archive writer, selecting one file among 16,000 sibling directories, reduced selection and archive creation from 8.09 s to 38 ms. This does not measure total CI runtime.

Validation: full unit suite with race detection and coverage, lint, and CLI build passed. Tests verify byte-for-byte archive equality, shared dependencies, cache reuse, literal/glob paths, renderer guards, annotation whitespace, and error cleanup. A benchmark covers 1,000–16,000 sibling directories.

ak-a and others added 4 commits November 2, 2026 13:57
Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Codex <noreply@openai.com>
Use literal exclusion lookups and skip directory metadata reads before writing tar entries. Preserve archive bytes and keep the Argo CD compressor for full-checkout fallbacks.

Co-Authored-By: Codex <noreply@openai.com>
@dag-andersen

Copy link
Copy Markdown
Owner

@ak-a Thank you for the PR! Unfortunately, I don't have time to review it before KubeCon next week :)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants