fix(master): stop MergeWal starving Compact in the scheduler

[beinan]

Problem

Compact tasks could sit queued indefinitely whenever MergeWal load was present. In production the 600s auto-sweep kept ~50–80 MergeWal tasks queued/running, fragments grew to 1,085,065 (1–10 rows each), and because every worker's open Dataset handle keeps the full manifest resident, 18 of 20 workers were OOMKilled. Scaling the master 3→8 replicas did not help — every added slot was consumed the same way.

Root cause: admission, not concurrency

MERGE_WAL_CONCURRENCY already gave merge its own semaphore, but a single dispatch loop called one unfiltered claim_next() and only chose a pool after the claim.

Claiming is destructive — it deletes the queue key, grants a lease, and takes the per-experiment target lock. So a claim made because the general pool had permits could return a MergeWal, then park it on the saturated merge semaphore while holding its target lock. With MergeWal numerically dominant, nearly every claim returned one, and the loop's while guard stayed true only because the general pool was idle. The dispatcher spun claiming work it could not run while a lone Compact sat queued behind it.

One correction to the issue's diagnosis

The issue lists "single shared queue, interleaved only by queue key order" as root cause #1. That part is not accurate: task ids are UUIDv7 (lance-context-core/src/id.rs), so the queue is genuinely time-ordered FIFO — ordering was never arbitrary.

This matters, because plain FIFO turns out to be sufficient once admission is fixed. No priority, fairness policy, or reserved capacity was added — those would solve a problem that isn't there.

Fix

Add TaskKinds, a small kind set threaded into claim_next, and run one poller per pool — GENERAL (Compact + IndexId) and MERGE_WAL — each claiming only the kinds it can actually run. The filter is applied before the dependency probe, so skipped kinds cost nothing.

FIFO order within a kind is unchanged, and no new config knob is introduced.

One caveat, now documented in the config docs rather than left implicit: MERGE_WAL_CONCURRENCY=0 shares a single pool, which necessarily reinstates single-poller behavior. Prefer a non-zero value when both kinds are in play.

Tests

Both etcd-backed, and both verified to fail with the fix reverted:

• filtered_claim_reaches_compact_behind_merge_wal_backlog — 30 MergeWal enqueued ahead of one Compact (worst case for FIFO). Asserts an unfiltered claim still returns MergeWal, so the test cannot silently stop exercising the scenario; that GENERAL reaches the trailing Compact; that GENERAL never returns MergeWal; and that the merge pool still drains its own backlog.
• compact_runs_while_merge_wal_pool_is_saturated — end-to-end through the dispatcher against a stub worker that accepts and never responds. Asserts the Compact reaches Done and that MergeWal work is still outstanding, so it cannot pass merely because the backlog drained.

Negative verification: with the kind filter removed, the first fails on the kind assertion and the second hangs to timeout — the production symptom exactly.

Full suite green: 22/22 etcd-backed master tests, 222 core, 68 server, fmt + clippy -D warnings clean.

🤖 Generated with Claude Code