Repository navigation
perf(scan): read the album-covers pass's stored images in one batch - #755
Merged
Merged
Conversation
After the walk, finish_album_artwork checks each album the scan saw still stores the cover the walk gave it, and skips the album when it does. It read each album's image rows with its own statement, on every scan and for unchanged albums too: one read per album, thousands on a large music library (gl-1277). The albums the pass could skip (not waiting for a track backfill, with a walk cover) are now read in one get_image_infos_batch, which splits the ids into chunks itself, and the pass skips the ones whose stored Primary is still their walk cover. Every other album has its cover published by inherit_album_cover as before. preload_album_artwork's batch could not be reused: it runs before the walk, which writes album rows after it. The pass serves the priority lane between albums, and the old per-album read came after it. A lane refresh that touched anything now drops the set, and the remaining albums are read again, so no album is judged by rows a served refresh may have rewritten. A failed read is logged and sends the albums to inherit_album_cover, as a failed single read did. The new library_scan_album_artwork_statements test pins a rescan of three unchanged albums to no single-album image read (three before) and checks every album and track keeps its image. Signed-off-by: Ken Smith <ken@mangoleafstudios.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A library scan's album-covers pass now reads its albums' stored images in one batched read instead of one statement per album. It runs on every scan, unchanged albums included. Which albums keep their cover and which have it published again is unchanged, and so are the images stored.
Upgrade note: none
Why
gl-1277. After the walk,
finish_album_artworkchecks each album the scan saw still stores the cover the walk gave it, and skips the album when it does. It read each album's image rows withget_image_infos(id), one statement per album on every scan. That is thousands of point reads on a large music library, each repeated on every routine rescan. The batched read the issue points at (preload_album_artwork's) runs before the walk, and the walk writes album rows after it, so it cannot be reused: the pass needs a fresh read of its own.finish_album_artworkis Ferrofin's own closing pass (it publishes the shared, content-addressed album cover the trackAudioImageProviderextraction gives its album, and backfills the tracks). It is not a port of one C# method. The change keeps every decision it makes, so nohandlers::VERIFIEDorVERIFIED_SUBSYSTEMSrow changes.How
crates/ferrofin-core/src/library_scan.rs:albums_keeping_their_coverreads the image rows of the albums the pass could skip (not inalbums_needing_backfill, with a walk cover) in oneget_image_infos_batch. The repository already splits the ids into chunks (padded_in_chunks). It returns the albums whose stored Primary is still their walk cover.finish_album_artworkbuilds that set on its first album that could be skipped and skips the albums in it, as the per-album check did. Every other album goes toinherit_album_coveras before.serve_lanereports that a refresh touched anything, the set is dropped and the remaining albums are read again in one batch, so no album is judged by rows a refresh may have rewritten.Verification
crates/ferrofin-core/tests/library_scan_album_artwork_statements.rs,a_rescan_reads_no_album_images_one_album_at_a_time. It records the statements a rescan of a three-album library runs. Before the fix:SELECT * FROM "BaseItemImageInfos" WHERE "ItemId" = ?1, one per album. After the fix it runs none, and every album and track keeps the same image path.cover.jpgand two FLAC tracks per album, remote fetchers off), created withPOST /Library/VirtualFoldersand rescanned twice withPOST /Library/Refresh.GET /Items/{id}/Imagesfor every album and track returned the same rows after all three scans (Primary, the shared album cover, 64x64, the same blurhash). In the server's sqlx debug log, a rescan's album-covers pass now makes one batchedBaseItemImageInfosread and no single-album read.cargo nextest run -p ferrofin-core: 5123 passed.cargo clippy -p ferrofin-core --all-targets --all-features -- -D warningsandcargo fmt --all --check: clean.08d0ebe9: fmt, clippy, doctests and ferrofin-core coverage (95.70 % lines) pass. nextest: 16529 of 16530 pass. The one failure isferrofin-wasm::wasm_hello_guest, which fails locally before it builds the guest (sccache: error: path must be shorter than SUN_LEN, the local sccache socket path). That is an environment problem, not this change. GitHub CI on the same head passes all 14 checks,wasm_hello_guestincluded.ferrofin-core(library scan closing pass). Performance is benchmarked per release.release/v1.5.0. No open PR changesfinish_album_artwork(pr_stack.py plan).Checklist
type(scope): subject) and are signed off (git commit -s)cargo fmt --all --checkcleancargo clippy --all-targets --all-features -- -D warningscleancargo nextest run --workspace+ doctests pass (locally, all but the sccache-boundwasm_hello_guest; GitHub CI runs that one green)UPSTREAM_TAG, and the operation'shandlers::VERIFIEDrow or the subsystem'shandlers::VERIFIED_SUBSYSTEMSrow added or updated with its divergences (a re-checked row also leaves its earlier-pin list), or the description says why the touched behaviour is unchangedferrofin-core,ferrofin-db,ferrofin-api) was touched: noted in the description (performance is benchmarked per release, not per PR)