feat(api): image map endpoints with cached UMAP projection and DBSCAN clusters by lstein · Pull Request #37 · invoke-ai/InvokeAI-7

lstein · 2026-08-02T20:51:51Z

Summary

PR 4 of the Image Map stack (on the deps PR). Adds the serving layer:

Pure projection math (image_index/projection.py, lazy imports so the numba JIT never slows startup): PhotoMapAI-tuned UMAP (n_neighbors<=15, min_dist 0.05, cosine, seeded); 2-3-point galleries fall back to a PCA projection (umap-learn cannot fit them); per-request DBSCAN with eps clamped relative to the projection's coordinate span and a hard point cap — both bound sklearn's O(pairs) neighborhood memory.
Projection jobs on the index worker: deduplicated per user (last request wins, so admin promotions/demotions can't pin a stale scope), run when the index is quiescent, cached per (user_id, model_id). A failed fit caches an empty projection under the current scope hash — clients see "empty" instead of hot-looping recompute requests, and the next gallery change retries naturally.
/api/v1/image_map router:
- GET /points?eps= — serves the cached projection, never blocks on a UMAP fit; clustering + response assembly run off the event loop (asyncio.to_thread). Stale caches are filtered against the caller's current accessible set before clustering, so un-shared/deleted images neither appear nor influence visible cluster labels via density-chaining.
- POST /refresh (202), GET /status (index counts admin-only; projection point_count filtered to the current scope).
New image_map_projection_ready socket event routed to user:{id} + admin rooms in a single deduped emit.

An adversarial review of this diff was run; both HIGH findings (2-3-image permanent failure loop; event-loop-blocking DBSCAN with measured OOM at 30k points/eps=2.0), the pre-filter clustering leak, duplicate admin emits, scope-pinning dedup, and the stale point-count leak are all fixed above with regression tests. Accepted trade-offs (documented): projection fits are single-threaded (seeded) and can occupy the worker for large galleries; /points is unpaginated.

Testing

59 tests: projection math (blob recovery, seed determinism, tiny-gallery PCA fallback, eps clamp, size cap), worker projection jobs (compute+cache+event, failure self-healing, dedup semantics), router auth matrix (no cross-user name/coordinate/count leakage; hidden points cannot influence visible clusters), staleness flips, live re-clustering.

🤖 Generated with Claude Code

… clusters Adds the serving layer for the semantic image map: - Pure projection math in `image_index/projection.py` (lazy imports so the numba JIT never slows app startup): PhotoMapAI-tuned UMAP (n_neighbors<=15, min_dist 0.05, cosine), per-request DBSCAN, and an order-insensitive scope hash fingerprinting the accessible image set. - Projection jobs on the index worker: deduplicated per user, run when the index is quiescent, cached per (user_id, model_id), and announced with a new `image_map_projection_ready` event routed to the user's socket room (plus admins). - New `/api/v1/image_map` router: - `GET /points?eps=` serves the cached projection with live DBSCAN clustering (eps adjustable without recompute) and never blocks on a UMAP fit; missing/stale caches enqueue a recompute. Stale caches are filtered against the current accessible set so un-shared or deleted images never leak out of an old projection. - `POST /refresh` (202) and `GET /status`. - Admins get the all-images scope; regular users get their accessible set. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…BSCAN eps budget Adversarial review of this PR against the merged indexer. **Projections starved by a systemic embedding outage.** The worker only reached projection work in the `batch is None` fall-through. The merged indexer deliberately charges no attempt for a systemic failure, so the same batch is returned on every pass, `batch is not None` stays true, and the projection branch became unreachable for the whole duration of an outage — /points reporting "computing" forever over a gallery that is in fact embedded. Neither side is wrong alone; the interaction is new. A projection reads only already-stored embeddings and needs no encoder, so the failed-batch path now falls through to it when the failure is systemic, while per-image failures still `continue` (they retire after _MAX_ATTEMPTS, so quiescence arrives on its own). **Projection requests dropped on any error outside the fit.** The job is popped from the dedup map before the work runs, but only `compute_umap` was inside the try. A raise from the DB reads or from `set_projection` ("database is locked" is a condition this code designs for elsewhere) unwound to the generic worker handler, which knows nothing about projections: /refresh had already answered `enqueued: true` and no `image_map_projection_ready` would ever fire. Both are now guarded and re-queue the job. **The eps floor overrode the memory budget it had just computed.** The 0.01 floor was applied *after* `_shrink_eps_to_pair_budget`, re-inflating eps past the neighbor-pair bound: a near-coincident map reached 400M pairs against a 50M budget, +3.8GB RSS at 20k points. The floor now applies before the shrink, the shrink iterates to a hard minimum instead of giving up after 12 rounds, and compute_clusters skips clustering when the budget still cannot be met — the same response shape as the existing point cap. Measured at 20k coincident points: 3.8GB -> 91MB. **Non-finite coordinates cached forever.** A NaN reaching the cache made /points raise inside sklearn on every request, and the cache is only replaced when the scope hash changes, so it 500ed that user indefinitely. Treated as a failed fit, which takes the existing empty-cache path. **Unchanged scopes recomputed.** Every accepted request re-ran the fit even when nothing it depends on had changed. Since the fit is seeded that burned minutes of single-threaded worker CPU to produce identical coordinates, and a client refetching on `projection_ready` could drive it in a loop. Short-circuited on the scope hash, still emitting so a refresh is acknowledged. Two of the new tests cover code that was already correct but unguarded: that `_next_projection_job` removes the job (leaving it turns the worker into an infinite recompute loop) and that the failed-batch path uses the escalating backoff rather than a fixed poll — the line most likely to be silently reverted by a hand-resolved rebase conflict, which is how this PR was rebased. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The scope-change phase of test_unchanged_scope_does_not_recompute_the_projection runs at 4 points — past compute_umap's small-N PCA fallback — so its wrapped real fit JIT-compiles numba on first call. The test's final wait returns on fit ENTRY, so stop()'s 10s join expires mid-compile and abandons the worker thread; pytest then moves on to test_projection.py, whose own main-thread fit runs concurrently with the abandoned one and aborts the interpreter (SIGABRT, observed on macOS CI). Count fits with a stub instead — the test asserts whether the fit runs, never what it produces — and wait for the stored row before teardown so stop() always joins an idle worker. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Adversarial review of this PR as a whole — three fresh-context reviewers over the projection math, the worker integration and the HTTP surface. The router's access control survived every attack (no cross-user leak through names, counts, labels, timestamps or cache keys), but the commit that fixed the LAST review introduced a worse bug than any it removed. **A failed fit made the map permanently empty.** Two deliberate behaviours combine into a dead end: the failure path caches an empty projection stamped with the CURRENT scope hash, and the new unchanged-scope short-circuit refuses to recompute while that hash matches. So one transient failure — a MemoryError while a generation holds RAM, a JIT hiccup, or the non-finite guard added in that same commit — leaves /points answering `state: "empty", stale: false` forever: honest-looking fields that stop the client asking, over a gallery that is fully embedded. Nothing recovers it. Not restarting (the row is in SQLite), and not POST /refresh, which the worker short-circuits too — and which no frontend code calls in any case. Only adding or deleting an image clears it. Before the short-circuit existed, /refresh recomputed and the failure healed. Failed scopes now get one more fit per process: a transient failure recovers on the next request, a permanent one costs one wasted fit per restart rather than one per poll, and /points asks for that retry since staleness never will. **A generation starved projections for its whole duration.** The projection branch sits after `_wait_for_idle_generation()`, so a single pending image parked the worker for the length of a run and /points reported "computing" throughout. A projection reads stored embeddings only — no encoder, no GPU — which is the same reasoning already used to let projections through a systemic embedding outage; this is that starvation reached by its common trigger. Every projection test builds its invoker with `device='cpu', session_queue=None`, where the wait returns immediately, which is why the suite never saw it. **A partial batch erased the outage it had just recorded.** `if stored: self._systemic_failures = 0` ran in a `finally`, i.e. after the handler that counted the failure. A batch whose first write landed and whose second hit "database is locked" — the contention this code designs for — reset the counter, flattening the escalating backoff to a 1 Hz retry and, since the projection fall-through keys on that counter, skipping projections for the whole window. Both existing systemic-failure tests fail every write, so `stored` is always empty and the branch was unreachable. **The worker began multi-minute fits after stop().** Neither path reaching the projection branch re-checked the stop flag (the backoff wait returns instantly once it is set), and _process_projection checked it nowhere, so a fit could start after stop() and write the database and emit after stop() returned. This is the root cause of the abandoned-worker SIGABRT that 4569791 worked around in a test. **The reported cluster_eps was not the eps DBSCAN used.** The router resolved eps to report it, then compute_clusters resolved it again — re-running the k-distance fit and the KD-tree budget shrink (~65% of a 50k-point request) and re-applying the 0.01 floor, which moves a budget-shrunk eps onto a different shrink grid. Split out cluster_at_eps so the reported value is the used value. Also: the embedding matrix is now read after the short-circuit rather than before, so a no-op refresh no longer materializes every accessible embedding (~307MB at 100k images); non-finite cached rows are filtered from the response and clustering failures degrade to unclustered instead of 500ing that user on every request until their gallery changes; a failed emit no longer unwinds into the backfill re-sweep; and `stale` is no longer reported true when nothing was actually enqueued. Each new test fails against the code as it stood: verified by stashing the service and watching all four go red. Reviewed but NOT changed, deliberately: the span clamp was reported as blanking small galleries, but at the default min_samples=10 it never alters the outcome (measured N=30..225, identical labels clamped and unclamped) — the small-gallery weakness is that min_samples does not adapt to N, and DBSCAN cannot form a cluster from fewer points than that. Retuning clustering on synthetic evidence would be a worse trade than leaving it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Both endpoints were unbounded work on shared resources, found while reviewing this PR. Neither needs a new field or status code. **POST /refresh.** Dedup collapses concurrent requests per user but not sequential ones, so a loop pinned the single index worker for every user and fanned an image_map_projection_ready event into every connected admin's socket per iteration. A recompute takes minutes on any gallery where this matters, so refusing a second request inside a 10s window costs a caller nothing. Throttled requests answer `enqueued: false`, which already means "not accepted", so no client change is required. Two placement details are load-bearing. The throttle is in the route, NOT in `request_projection`: /points calls that same method to trigger the one-shot retry of a failed projection, and gating it there would make a failed fit permanent again — the bug the previous commit fixed. And a request that was throttle-admitted but then refused by the service (the indexer is not running) releases its slot, so a client polling across a restart is not locked out of the first refresh that could actually succeed. **GET /points.** The endpoint is polled, and between polls nothing it clusters has usually changed, so it repeated identical DBSCAN work: measured 0.68s and 103MB per request at the 50k-point clustering cap, or 1.21s and 238MB at the documented min_samples ceiling, on the shared default executor. Labels are now memoized on every input that can change them — the projection row (its scope hash plus updated_at, which moves on each rewrite), the caller's currently accessible subset, eps, and min_samples — bounded to 8 entries, which is a few MB since an entry is one int64 label array. user_id is in the key as well, so a collision on every other component still cannot serve one user's labels to another; there is a test for that. The degraded all-noise fallback is deliberately not cached, or a one-off clustering failure would stick. Both caches prune: the throttle map drops entries older than the window on write rather than keeping one row per user who has ever refreshed, and the label cache evicts least-recently-used past its bound. Verified the tests pin behaviour rather than existence: with the interval set to zero and the cache read neutered, the throttle and reuse tests fail on their own assertions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Adversarial review of the previous fix commit, which removed one terminal state and introduced two unbounded ones. The reviewer's central charge is fair: a commit written to correct a review is where the next bug goes. **The retry became an endless request/event cycle.** /points asked for a recompute on every poll where the cached projection had no points, and the worker's short-circuit emits image_map_projection_ready unconditionally — an event the frontend answers with another /points. So once a scope's single retry was spent, each poll queued a job that did nothing except emit, which triggered the next poll, at the worker's rate for the life of the process, each turn costing two accessible-set scans and two SHA-256 passes over every image name. request_projection now takes the failed scope and refuses once its retry is spent, so the cycle stops at the source and the state settles into an honest "empty" instead of a permanent spinner. **Projections ran between every backfill batch.** Moving the projection call ahead of the generation wait put it on every worker pass that had embedding work, not just the passes that were about to park behind a generation — so a client watching the map during the initial index build drove one full UMAP fit per batch, each obsolete the moment the next batch stored, which is exactly why projections are otherwise deferred to quiescence. Now gated on a generation actually being in progress: the case where the embed is about to block anyway and nothing new can enter the index, so the scope hash holds still and repeat requests short-circuit. **A transient error burned the retry without using it.** The budget was spent when the retry was granted, but two early returns — a failed embedding read and a stop during shutdown — sit between that point and the fit. A "database is locked", the most designed-for transient error in this file, therefore consumed the one retry with zero fits attempted and left the map blank in a process where it would have worked. The budget is now spent where the fit actually begins. **An all-non-finite row was still permanently blank.** The retry keyed on the cached point_count while the response filters non-finite coordinates, so a row with points but no *finite* points served "empty, not stale" with nothing ever asking for a recompute — the precise terminal state the retry exists to remove, reached through the guard added alongside it. The decision now keys on what is actually servable, computed once and shared with the response builder. **Reverted: the systemic-failure counter now resets on partial progress again.** Moving the reset off the `finally` fixed a batch that erased its own outage, but left no reset path at all while every batch partially failed, walking the retry wait to its 60s ceiling while the index was still storing an image per batch. Under mild write contention that is worse than the flat 1Hz retry it corrected: a batch that stores anything is making progress, the backlog drains, and quiescence arrives on its own. The test now pins that reasoning rather than the opposite. Also clears the retry markers and any requeued job on start(), so an in-process restart neither denies a scope the retry its docstring promises nor resurrects a shutdown-time job carrying an admin scope the user may no longer hold. Both new router tests fail against the previous behaviour: with the guards reverted the spent-retry test reports "computing" forever. Still open and deliberately not addressed here: /cluster_labels on the cluster-labels branch repeats the double-resolve and lacks the non-finite filter, so its labels can be computed at a different eps than /points reports. That belongs in the branch that adds it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…serve Review of the two preceding commits found the request/event loop they were written to kill still reachable, through the very case the second one added. /points decides "this projection failed" from what it can serve: it drops non-finite rows, so a row with points but no FINITE points is empty to every client. The service decided from point_count. For a row carrying point_count > 0 and all-NaN coordinates — what a database written before the projection writer grew its isfinite guard still holds — the two disagreed: /points asked for a retry on that row's behalf, the service granted the request without entering the retry branch, the worker short-circuited and emitted projection_ready anyway, and the client (which refetches on that event) asked again. Because the branch was never entered the budget was never spent, so the refusal that is supposed to break the cycle could never fire: a permanent request/emit loop at the worker's poll rate, and a permanent spinner, per affected user. The service now decides the same way /points does, on whether any finite point exists. Also from the review: - The budget was spent before the fit, on the argument that nothing between the spend and compute_umap can return. That ignored the write: a fit that SUCCEEDS and then loses its set_projection to a locked database re-queues, and the re-queued job found the old empty row with the budget already gone — minutes of correct work discarded and the map blank for good, without a single failed fit anywhere in the sequence. It is spent once the failed result is durably cached instead. - /refresh answered `enqueued: true` while the worker was guaranteed to short-circuit, so a spent budget was unrecoverable while the server ran. An explicit, user-initiated request now clears it. Deliberately not a bare request: /points' staleness path is driven by polling, and clearing there would hand a poller the unlimited retries the budget exists to deny. - The generation gate is gone; projections are serviced from inside _wait_for_idle_generation instead. A gate is read once per pass, so a request arriving after the worker had already parked waited out the whole run — the symptom it was added to fix. This also removes a latent hang in test_projection_does_not_wait_for_an_in_progress_generation, which only passed because the main thread reliably won a race against the worker's first pass. - /points hoisted its O(N) accessibility mask onto the event loop to decide the retry, contradicting the comment three lines below it: 54ms of event loop time per poll on a 500k-image gallery, stalling every other request in the process. The mask is back on the worker thread and the retry decision reads the one fact it needs from the result. - The cluster cache is one entry per user rather than a shared pool of 8. The pool was worse than no cache: with more concurrent map users than slots, strict LRU over their round-robin polling misses every time, and any single caller could evict everyone else's labels by varying `eps` a handful of times. It also stored the above-the-cap all-noise array, which is free to recompute and unbounded in size — the one entry the ~400KB bound did not cover. Only a clustering that actually ran is cached now. - /refresh releases its throttle slot if request_projection raises. The router's fake service cannot demonstrate any of the retry contract — it decides the refusal from the argument alone, with no view of the cached row — so the three regression tests here are service-level, and each was verified to fail against the behaviour it replaces. The cross-user cache test was vacuous for the same reason (the two users' keys already differed in three other components) and is now asserted on the cache helpers directly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

lstein requested review from JPPhoto and blessedcoolant as code owners August 2, 2026 20:51

github-actions Bot added python python-tests api services labels Aug 2, 2026

lstein marked this pull request as draft August 2, 2026 22:23

lstein force-pushed the feat/image-map-03-deps branch from 13a64aa to fdb7e82 Compare August 2, 2026 22:38

lstein force-pushed the feat/image-map-04-projection branch from 236d5ef to 5091e5d Compare August 2, 2026 22:38

github-actions Bot added the frontend label Aug 2, 2026

lstein force-pushed the feat/image-map-04-projection branch from 5091e5d to ec55e63 Compare August 3, 2026 01:38

lstein force-pushed the feat/image-map-03-deps branch from fdb7e82 to b86155a Compare August 3, 2026 01:38

lstein force-pushed the feat/image-map-04-projection branch from ec55e63 to 9a46659 Compare August 3, 2026 02:26

lstein marked this pull request as ready for review August 3, 2026 18:28

lstein requested a review from Pfannkuchensack as a code owner August 3, 2026 18:28

lstein force-pushed the feat/image-map-04-projection branch from 9a46659 to 589a09b Compare August 4, 2026 01:43

lstein force-pushed the feat/image-map-03-deps branch from b86155a to 3b0243b Compare August 4, 2026 01:43

lstein force-pushed the feat/image-map-04-projection branch from 589a09b to bbab55d Compare August 4, 2026 05:37

lstein force-pushed the feat/image-map-03-deps branch from 3b0243b to 99ff815 Compare August 4, 2026 05:37

lstein force-pushed the feat/image-map-04-projection branch from bbab55d to fe1b01c Compare August 4, 2026 14:58

lstein force-pushed the feat/image-map-03-deps branch from 99ff815 to 086926d Compare August 4, 2026 14:58

lstein force-pushed the feat/image-map-04-projection branch from fe1b01c to d1976a2 Compare August 4, 2026 22:32

lstein force-pushed the feat/image-map-03-deps branch from 086926d to 178f253 Compare August 4, 2026 22:32

lstein force-pushed the feat/image-map-04-projection branch from d1976a2 to 8754846 Compare August 5, 2026 00:31

lstein force-pushed the feat/image-map-03-deps branch from 178f253 to 41592b3 Compare August 5, 2026 00:31

lstein force-pushed the feat/image-map-04-projection branch 3 times, most recently from 8984cd4 to 0fc7c71 Compare August 7, 2026 20:41

lstein force-pushed the feat/image-map-03-deps branch 2 times, most recently from 3ecd588 to 03566e4 Compare August 8, 2026 23:33

lstein force-pushed the feat/image-map-03-deps branch from 03566e4 to 6822fbb Compare August 8, 2026 23:55

lstein force-pushed the feat/image-map-04-projection branch from 0fc7c71 to 46407d2 Compare August 8, 2026 23:56

Base automatically changed from feat/image-map-03-deps to main August 9, 2026 01:33

lstein force-pushed the feat/image-map-04-projection branch 2 times, most recently from 8326208 to 6214ee0 Compare August 13, 2026 04:36

lstein and others added 7 commits August 13, 2026 17:55

lstein force-pushed the feat/image-map-04-projection branch from 6214ee0 to 11c12a5 Compare August 13, 2026 23:17

lstein merged commit 32c875e into main Aug 13, 2026
19 checks passed

lstein deleted the feat/image-map-04-projection branch August 13, 2026 23:46

Navigation Menu

Repository navigation

feat(api): image map endpoints with cached UMAP projection and DBSCAN clusters - #37

feat(api): image map endpoints with cached UMAP projection and DBSCAN clusters#37
lstein merged 7 commits into
mainfrom
feat/image-map-04-projection

lstein commented Aug 2, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

lstein commented Aug 2, 2026

Summary

Testing

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant