Compact a quiet table's fresh files every minute, between sweeps (T-627)

[chasers]

TL;DR: Compaction now merges a quiet table's newest files every minute, not every 5 minutes. A recent-data query opens a few files instead of a dozen tiny ones.

Tracker: T-627. Stacked on #450 (T-626).

Why

• A table that seals on age, not size, leaves one sealed file per seal.
• On the sandbox, logs.cluster (about 110 rows a minute, 3 write partitions) sealed 2-3 files a minute of 1-30 rows each.
• The hour lane ran once per compact_interval_ms (5 min). So a dozen tiny files piled up in the current hour. 17 of 24 files held fewer than 1,000 rows.
• Every query over recent data opened each one over S3.
• A 60-minute histogram ending now took about 5x one ending 15 minutes ago, on the same row count (776 ms vs 146 ms median).

What changed

• New fresh tick in Smolquery.StorageService.Scheduler. Every compact_fresh_interval_ms (60 s) it runs the hour lane again for the quiet tables only.
• A quiet table has a file in the current or previous compact_bucket_ms under compact_fresh_below_bytes (4 MiB). New Planner.fresh_tables/3 decides this from the listing's bytes, with no footer reads.
• Each sweep refreshes the quiet set. A table whose listing is no longer quiet leaves the set.
• New env vars: SMOLQUERY_COMPACT_FRESH_INTERVAL_MS (0 = off) and SMOLQUERY_COMPACT_FRESH_BELOW_BYTES.
• Sweeper gains two optional callbacks for a second timer: on_start/1 and handle_tick/2. The retention sweeper defines neither, so it does not change.
• The lane timer reports lane="fresh".
• Docs: docs/configuration.md rows, and a docs/deployment.md upgrade note.

How it works

1. A sweep lists every table, as before, and computes the quiet set from the listings.
2. 60 s later, the fresh tick lists only the quiet tables.
3. For each one it plans the hour lane, the same planner as a sweep, but only over files under compact_fresh_below_bytes, and merges them into one group. A bigger file waits for the 5-minute sweep.
4. The outcomes update the row caps, the quarantine and the backoff, as a sweep's do.
5. The tick runs in the scheduler process, so it never races a sweep over the same files.

Why this option

You chose this option over:

• sealing quiet tables less often, which keeps rows in the hot tier longer and gives more hot segments per query
• per-table write partitions = 1, which is config only and does not fix the merge lag

Tests

• ✅ PlannerTest: fresh_tables/3 picks a table with a small current- or previous-hour file. It skips a table whose recent files are big, and a table whose small files are older. It returns nothing when the tick is off.
• ✅ SchedulerTest (integration, real DuckLake and merge): a sweep marks the quiet table; 2 new seals arrive; one fresh tick merges all 3 into one segment, and no rows are lost.
• ✅ SchedulerTest: a table whose recent files are not small stays out of the set.
• ✅ SchedulerTest (integration): a 20,000-row file over a 20 KB fresh floor stays; the tick merges only the 2 small seals.
• ✅ Mutation checks: with the fresh tick as a no-op, the merge test fails. Without the fresh floor, the floor test fails.

Review

A Fable review found no row loss and no broken invariant. The ownership gate, backoff, caps and quarantine behave as in a sweep. Fixed in Review of T-627: ...:

• Write amplification. The tick used the hour lane's 32 MiB floor, so it could rewrite a growing file every minute. Now it merges only files under compact_fresh_below_bytes (4 MiB).
• Spill gating. The tick logged and counted a "span paused" every minute and, when gated, planned settled files. Now it uses the un-gated runtime and does nothing without a quiet table.
• Tests. The integration test waited 5 s around a real merge; now 60 s. The fresh timer is off in test config, so no test gets a background merge.
• Docs. The moduledoc now says caps, quarantine and conflict waits advance per tick. The cost line no longer says "small by definition".

Checks

• ✅ mix precommit: 3,090 tests pass
• ✅ mix ci
• ✅ mix dialyzer

Watch out

• ⚠️ More swaps. Each quiet table can get one merge and swap per minute, of at most 4 MiB. Each swap can conflict with a seal of the same table (T-573). Watch smolquery_compaction_conflicts_total.
• ⚠️ More listings. One catalog listing per quiet table per minute. A table with many small files (like metrics.samples when backlogged) is listed each minute too.
• ⚠️ Caps, quarantine and conflict waits now advance per tick as well as per sweep. A lost commit parks the table for compact_interval_ms, skipping its ticks until then.
• ⚠️ The hot tier is unchanged. T-627's "<10 hot segments per Grafana selector" item is not addressed here.
• ⚠️ T-627 sandbox acceptance is open: at most ~5 sealed files under 1,000 rows on logs.cluster, and the 60-min histogram ending now within 1.5x of one ending 15 min ago.

🤖 Generated with Claude Code