Skip to content

Sync: speed up the full library sync and make it resumable#735

Open
RaulitoRS97 wants to merge 6 commits into
BLeeEZ:masterfrom
RaulitoRS97:sync/resync-performance
Open

Sync: speed up the full library sync and make it resumable#735
RaulitoRS97 wants to merge 6 commits into
BLeeEZ:masterfrom
RaulitoRS97:sync/resync-performance

Conversation

@RaulitoRS97

Copy link
Copy Markdown
Contributor

Full library sync on a 58k-song / 23.7k-album server: 35–45+ min → ~2 min on OpenSubsonic servers (Navidrome tested) and → ~10 min on any other server — cooler, resumable, and with no behavior change for servers that don't support the fast path.

Problem

Measured on a real library (58k songs / 23.7k albums, Navidrome, iPhone):

  • The background song sync issues one getAlbum request per album (~24k requests) strictly sequentially, and every album pays two background-context saves (~48k saves total). It ran at ~40 songs/s with CPU pinned at ~134%, Energy Impact "Very High", and the device heated up until iOS hardware-throttled it.
  • An interrupted initial sync restarts from zero on the next launch, and its fully parallel writes could create duplicate artists/albums.

Changes (one commit each)

  1. Make the initial sync resumable with bounded parallel album requests. The album phase fetches pages with a bounded window (4 in flight) while writes stay serialized, removing the duplicate-entity window at its source. Completed pages are checkpointed in the account settings (with an album-count fingerprint that discards stale checkpoints when the server library changed), so an aborted or killed initial sync resumes where it stopped. The pre-sync library wipe only runs when not resuming; an explicit "Resync Library" always starts fresh; Skip cancels the running sync task.

  2. Parallelize background song sync fetches and batch Core Data writes. The serial OperationQueue becomes a paged driver: song fetches run 4 in parallel, parsing/writes stay serialized and are batched — 16 albums per background context and save instead of 2 saves per album (~48k saves → ~1.5k). Cancellation is re-checked before every batch write and albums deleted in the meantime are skipped via existingObject, so a resync/logout can no longer receive late writes. sync(album:) now parses the response once into a single context (previously two performs re-parsed the same XML).

  3. Bound the newest-albums song fetch with the same machinery. The quick "newest albums" sync (pull-to-refresh, Home) used an unbounded task group — up to 50 concurrent writers, and one failure cancelled all of them. It now reuses the bounded/batched path.

  4. Respect thermal state and Low Power Mode. Between batches: serious thermal state → 2 parallel fetches, Low Power Mode → 1, critical → bounded pause until the device cools; transitions are logged. Measured: identical total duration on a cool device; when it heats up, the sync adapts its pace instead of running into OS hardware throttling.

  5. Add a bulk fast path via OpenSubsonic search3. The OpenSubsonic spec requires servers to support an empty search3 query "to allow clients to properly access all the media information for offline sync" — this implements exactly that: a paged empty-query walk (500 songs/page) fills all albums' songs in ~120 requests instead of ~24k. A small write-free probe detects support at runtime; servers that don't support it (classic Subsonic, Airsonic, Ampache) automatically keep using the per-album path from commit 2, so no setup is worse off. Albums are marked synced per page (confirmed once a later page contains no more of their songs), so progress stays gradual and partial progress survives cancellation. The bulk path deliberately performs no deletion reconciliation: a blank/truncated response (e.g. [Bug]: Since version 0.51, search3.view returns blank instead of JSON navidrome/navidrome#2862) is indistinguishable from the end of the listing and could mass-delete real songs.

  6. Apply the critical thermal pause to the bulk path (shared helper; the width throttling doesn't apply since the bulk walk is already sequential).

Measured results (58k songs / 23.7k albums, Navidrome, same device and network)

before after (per-album path) after (OpenSubsonic bulk)
Full song sync 35–45+ min ~10 min ~2 min
Throughput ~40 songs/s 200 songs/s (100 when hot, self-throttled) ~500 songs/s
CPU ~134% ~106% low
Energy Impact Very High High low
Thermals OS hardware throttling app-managed, stabilized stays cool
Interrupted initial sync restarts from zero resumes resumes

Verification

  • AmperfyKitTests pass; SwiftFormat (BuildTools/applyFormat.sh) applied — no reformatting needed.
  • Real-device testing: fresh full syncs with count verification against the server (albums/artists/songs match exactly), kill-and-relaunch mid-initial-sync (resumes), abort via airplane mode + resume, Resync Library and logout mid-sync, Skip, and Low Power Mode toggled mid-sync.

The tuning constants (4 parallel fetches, 16 albums per save, 500 songs per bulk page) are my best measured trade-offs on the hardware above — happy to adjust if you prefer different values.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant