Skip to content

perf(cleanup): avoid reading manifests for version retention - #9019

Open
ziwenzhang wants to merge 1 commit into
lance-format:mainfrom
ziwenzhang:spread-clean-up
Open

perf(cleanup): avoid reading manifests for version retention#9019
ziwenzhang wants to merge 1 commit into
lance-format:mainfrom
ziwenzhang:spread-clean-up

Conversation

@ziwenzhang

Copy link
Copy Markdown

Summary

  • Optimize CleanupPolicyBuilder::retain_n_versions to use Dataset::version_refs() instead of Dataset::versions(), avoiding reading and deserializing every manifest when only version numbers are needed.
  • Add a focused regression test that verifies the retention threshold remains correct and that retain_n_versions
    performs zero object-store reads.

Testing

  • cargo test -p lance retain_n_versions_does_not_read_manifests -- --nocapture
  • cargo test -p lance cleanup_and_retain_3_recent_versions -- --nocapture
  • cargo fmt --all -- --check
  • cargo clippy -p lance --lib --tests --benches -- -D warnings

@ziwenzhang

Copy link
Copy Markdown
Author

cc @zhangyue19921010 would you mind taking a look when you have a chance? Thanks!

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

The retention cutoff depends only on ordered attached version IDs, so reusing the lightweight version-reference listing preserves the existing branch, gap, and detached-manifest semantics while removing per-manifest payload reads. The focused regression covers both the cutoff and that I/O contract.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

K-approved Latest Gatekeeper recommendation permits acceptance. performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant