[QuintinBotes]

Part of #1300: the claim order. The reserved lane follows as its own PR, as agreed there.

Why

claim_one ordered only by run_at, so every kind waited in one queue. While hundreds of documents wait for extraction, the hourly sync_source and the process_document jobs for newly synced files wait behind every older extract_document. On a local model that was days: a source went unsynced for ten hours, and 423 new documents stayed pending and unsearchable.

What changes

• Migration 0116 adds job_priority(kind). It returns 0 for the kinds that make no chat call and 1 for the rest. Those kinds are sync_source, process_document, hydrate_rss_entry, embed_ontology, memory_ingest, build_vector_index, materialize_inferences and noop: the list from A sync or a new document waits behind the whole extraction backlog: jobs are claimed strictly by run_at #1300, with memory_ingest added as you suggested. The priority comes from kind, so no enqueue site changes. It also adds the index jobs_priority_claim_idx on (job_priority(kind), run_at) WHERE status = 'queued'.
• claim_one orders by job_priority(kind), run_at. With an empty jobs table and enable_seqscan off, EXPLAIN of the claim query shows an index scan on the new index and no sort node.
• A function, not a column. In A sync or a new document waits behind the whole extraction backlog: jobs are claimed strictly by run_at #1300 I said I'd add a priority column. A stored column would rewrite all of jobs, finished jobs included, during the startup migration. An IMMUTABLE function with an expression index gives the same order and index without that. The list lives only in the function, and the migration's comment says how to change it: CREATE OR REPLACE in a new migration and rebuild the index.
• Migration number. It is 0116, after the 0115 that Finding a base's name facts without vectors reads only its name facts #1318 brought. CURRENT_SCHEMA_VERSION is 104, the number of files in migrations/.
• Test. jobs_claim_tests.rs checks that a sync_source queued later is claimed before an earlier extract_document. Both rows are dated a century back, so claim_one takes them before anything else in the database, and the test deletes them afterwards. Without the ORDER BY change it fails with left: Some("extract_document").

#1316 by @linhongyu510 takes the same approach in claim_one; this PR adds the index and the claim test discussed in #1300.

How it was checked

On an M1 Max (macOS), at the PR head on top of dev 0daaed0:

• cargo fmt --all --check: passes.
• cargo clippy --workspace --all-targets -- -D warnings: no issues.
• cargo test --workspace --no-fail-fast with UTOPIA_DATABASE_URL pointing at a database created and migrated for this branch alone: 1,641 passed, 0 failed, 6 ignored.
• The new test with UTOPIA_TEST_REQUIRE_DB=1: it passes, and fails without the ORDER BY change.

Before review

• Every commit is signed off (git commit -s)
• cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings and cargo test --workspace pass
• SQL under crates/utopia-store/ was tested with UTOPIA_DATABASE_URL set (those tests skip without it)
• A new migration takes the next free number on dev, and CURRENT_SCHEMA_VERSION in crates/utopia-cli/src/main.rs equals the number of files in migrations/

[WaylandYang]

Thank you, Quintin. This is exactly the shape we discussed in #1300, and the description made it easy to review. Using an IMMUTABLE function with an expression index instead of a stored column is a good call: same order, same index, without rewriting all of jobs at startup, and the migration comment tells the next person how to change the list. All eight kinds check out against dispatch (none makes a chat call), and the claim test is well isolated and fails without the ORDER BY change. #1300 stays open for the reserved lane. Merging.