Sync: speed up the full library sync and make it resumable#735
Open
RaulitoRS97 wants to merge 6 commits into
Open
Sync: speed up the full library sync and make it resumable#735RaulitoRS97 wants to merge 6 commits into
RaulitoRS97 wants to merge 6 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Full library sync on a 58k-song / 23.7k-album server: 35–45+ min → ~2 min on OpenSubsonic servers (Navidrome tested) and → ~10 min on any other server — cooler, resumable, and with no behavior change for servers that don't support the fast path.
Problem
Measured on a real library (58k songs / 23.7k albums, Navidrome, iPhone):
getAlbumrequest per album (~24k requests) strictly sequentially, and every album pays two background-context saves (~48k saves total). It ran at ~40 songs/s with CPU pinned at ~134%, Energy Impact "Very High", and the device heated up until iOS hardware-throttled it.Changes (one commit each)
Make the initial sync resumable with bounded parallel album requests. The album phase fetches pages with a bounded window (4 in flight) while writes stay serialized, removing the duplicate-entity window at its source. Completed pages are checkpointed in the account settings (with an album-count fingerprint that discards stale checkpoints when the server library changed), so an aborted or killed initial sync resumes where it stopped. The pre-sync library wipe only runs when not resuming; an explicit "Resync Library" always starts fresh; Skip cancels the running sync task.
Parallelize background song sync fetches and batch Core Data writes. The serial OperationQueue becomes a paged driver: song fetches run 4 in parallel, parsing/writes stay serialized and are batched — 16 albums per background context and save instead of 2 saves per album (~48k saves → ~1.5k). Cancellation is re-checked before every batch write and albums deleted in the meantime are skipped via
existingObject, so a resync/logout can no longer receive late writes.sync(album:)now parses the response once into a single context (previously two performs re-parsed the same XML).Bound the newest-albums song fetch with the same machinery. The quick "newest albums" sync (pull-to-refresh, Home) used an unbounded task group — up to 50 concurrent writers, and one failure cancelled all of them. It now reuses the bounded/batched path.
Respect thermal state and Low Power Mode. Between batches:
seriousthermal state → 2 parallel fetches, Low Power Mode → 1,critical→ bounded pause until the device cools; transitions are logged. Measured: identical total duration on a cool device; when it heats up, the sync adapts its pace instead of running into OS hardware throttling.Add a bulk fast path via OpenSubsonic
search3. The OpenSubsonic spec requires servers to support an emptysearch3query "to allow clients to properly access all the media information for offline sync" — this implements exactly that: a paged empty-query walk (500 songs/page) fills all albums' songs in ~120 requests instead of ~24k. A small write-free probe detects support at runtime; servers that don't support it (classic Subsonic, Airsonic, Ampache) automatically keep using the per-album path from commit 2, so no setup is worse off. Albums are marked synced per page (confirmed once a later page contains no more of their songs), so progress stays gradual and partial progress survives cancellation. The bulk path deliberately performs no deletion reconciliation: a blank/truncated response (e.g. [Bug]: Since version 0.51, search3.view returns blank instead of JSON navidrome/navidrome#2862) is indistinguishable from the end of the listing and could mass-delete real songs.Apply the critical thermal pause to the bulk path (shared helper; the width throttling doesn't apply since the bulk walk is already sequential).
Measured results (58k songs / 23.7k albums, Navidrome, same device and network)
Verification
AmperfyKitTestspass; SwiftFormat (BuildTools/applyFormat.sh) applied — no reformatting needed.The tuning constants (4 parallel fetches, 16 albums per save, 500 songs per bulk page) are my best measured trade-offs on the hardware above — happy to adjust if you prefer different values.