[linhongyu510]

What

Fixes #1300.

claim_one ordered queued jobs by run_at alone. Every job kind shares that one queue. With hundreds of extract_document jobs backed up behind a slow local chat model, an hourly sync_source and the process_document jobs for newly synced items waited until the entire LLM backlog drained — on a local model that meant 10+ hours. New documents stayed pending and unsearchable the whole time.

Fix

Adds a two-tier ordering in claim_one: the three intake kinds that make no chat call (sync_source, process_document, embed_ontology) are claimed before LLM work. Within each tier, run_at still decides — so intake jobs are FIFO among themselves, and extraction jobs are FIFO among themselves.

This is the first of the three approaches listed in #1300, without a schema migration:

ORDER BY
  CASE WHEN kind IN ('sync_source', 'process_document', 'embed_ontology')
       THEN 0 ELSE 1 END,
  run_at

Why not a priority column

A priority column would be more explicit, but requires a migration and changes to every enqueue call site. The CASE is a one-line change that solves the reported starvation immediately. If you prefer the column approach, I'm happy to do that instead.

What this does NOT do

• Does not create a separate worker lane — all jobs still share the same worker pool.
• Does not change retry/backoff behavior.
• Does not prioritize build_vector_index, materialize_inferences, or other maintenance jobs — those don't block searchability of new material.
• Does not starve extraction: once intake jobs drain, LLM jobs resume in run_at order.

Verification

• cargo check -p utopia-store passes.
• The existing pure-function tests in jobs.rs (retry_delay, deferred_retry_secs) are unchanged.
• An integration test with PostgreSQL would need the CI environment; I verified the SQL ordering logic manually against the existing FOR UPDATE SKIP LOCKED pattern.

[WaylandYang]

Thank you, @linhongyu510, for jumping in on #1300 and for the clear write-up. The diagnosis is right, and it's great to have a first contribution from you. A few things before this can land:

1. Base and DCO. Please retarget the PR to dev (it currently targets main and is based on a commit from 10-11), and sign off the commit (git commit --amend -s, then force-push). We can't merge without the DCO line.
2. The claim has to stay index-backed. Today claim_one walks jobs_claim_idx (run_at) and stops at the first unlocked row. Nothing indexes ORDER BY CASE kind …, run_at, so every claim would read and sort the whole due backlog to take one row; during a full-text rebuild that's tens of thousands of rows per claim, on every worker, exactly when the queue is longest. The fix needs no call-site changes: a migration (0115 is the next free number) adding priority smallint GENERATED ALWAYS AS (CASE WHEN kind IN (…) THEN 0 ELSE 1 END) STORED, an index on (priority, run_at) WHERE status = 'queued', and ordering the claim by priority, run_at.
3. A test. One DB-backed test (with the utopia_store::test_db::url() guard) showing a later sync_source is claimed before an earlier extract_document. CI runs it against Postgres.
4. Please change "Fixes #1300" to "Part of #1300". Extraction jobs keep their worker slots while waiting for the model permit, so ordering alone doesn't get a sync through when every worker is parked; #1300 still needs the reserved lane.

One coordination note: @QuintinBotes said on #1300 yesterday that they're picking this up as a priority PR followed by a lane PR. Could you two agree there on who does which part, so nobody's work is wasted? Either way, thank you, and we're happy to review the next round.

[WaylandYang]

Thank you, @linhongyu510, for jumping on this so quickly; your ordering in claim_one was the right idea. #1324 has now landed with the same ordering plus the index and claim test agreed in #1300, so I'm closing this one. A review of the reserved-lane follow-up on #1300 would be very welcome, and we'd be glad to see your next contribution.