Skip to content

perf(metadata): bound locator compaction pressure #608

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
forhappy wants to merge 1 commit into from
Closed

perf(metadata): bound locator compaction pressure #608

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions crates/crab-metadata/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -403,6 +403,13 @@ covered by their pinned repository snapshot, then open the named checkpoint for
the actual lookup. A missing marker means publication is incomplete; malformed
or name-mismatched marker content fails closed.

Locator maintenance bounds the shared L0 frontier and compaction fan-in at 16.
The coordinator, worker and result commit poll every 500 ms while maintenance
is enabled. This prevents frequent checkpoint flushes from starving sorted-run
compaction or waiting on the dependency's five-second coordinator default.
Publication enables maintenance under measured L0 pressure; coverage-only and
reader-repair writers keep their existing maintenance-free behavior.

### Snapshot identity

`RepositorySnapshot` also captures the validated canonical layout descriptor.
Expand Down
11 changes: 9 additions & 2 deletions crates/crab-metadata/src/git_object_locator/writer.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,10 @@ const DB_LABEL: &str = "git_object_catalog_db";
const MAX_BATCH_ROWS: usize = 25_000;
const MAX_BATCH_LOGICAL_BYTES: usize = 2 * 1024 * 1024;
const LOCATOR_L0_SST_BYTES: usize = 64 * 1024 * 1024;
const LOCATOR_L0_MAX_SSTS: usize = 32;
// The scheduler favors L0 until similarly sized sorted runs fill this fan-in.
// Bound both frontiers so frequent checkpoint flushes cannot accumulate dozens
// of runs while new L0 work continually takes the sole compaction slot.
const LOCATOR_L0_MAX_SSTS: usize = 16;
const LOCATOR_COMPACTION_TRIGGER_SSTS: usize = LOCATOR_L0_MAX_SSTS / 2;
const LOCATOR_COMPACTION_MAX_CONCURRENT: usize = 1;
const LOCATOR_COMPACTION_MAX_SUBCOMPACTIONS: usize = 1;
Expand Down Expand Up @@ -1596,6 +1599,10 @@ fn should_start_compactor(l0_ssts: usize, planned_object_rows: u64) -> bool {
fn locator_settings(compact: bool) -> Settings {
let compactor_options = compact.then(|| {
let mut compactor = CompactorOptions {
// Flush backpressure waits for coordinator scheduling, not worker
// polling. Keep all three phases aligned so a full frontier does
// not idle for SlateDB's five-second default coordinator tick.
poll_interval: std::time::Duration::from_millis(500),
commit_compacted_interval: std::time::Duration::from_millis(500),
max_concurrent_compactions: LOCATOR_COMPACTION_MAX_CONCURRENT,
..CompactorOptions::default()
Expand Down Expand Up @@ -1969,7 +1976,7 @@ mod tests {
.expect("locator compactor enabled");
assert_eq!(
compactor.poll_interval,
CompactorOptions::default().poll_interval
std::time::Duration::from_millis(500)
);
assert_eq!(
compactor.commit_compacted_interval,
Expand Down
Loading