Summary
Client commit memory grows roughly linearly with the size of the committed batch (~1.5x batch bytes peak). Seeding a large existing tree as one commit exhausts a 128 GB workstation; splitting the same content into small batches completes fine with bounded memory. Expected: commit streams fragments with backpressure, keeping memory roughly constant regardless of commit size.
Environment
- Client:
lore 0.8.5+312, Windows 11 Pro for Workstations (Ryzen 9 7950X, 128 GB RAM)
- Server:
loreserver 0.8.5 built from source (lore-server/Dockerfile, linux/amd64), Docker on a Synology DS1821+, local filesystem stores
- Repo working tree: 648k files / ~610 GB on an iSCSI-mounted NTFS volume (~1.5 Gbps during commit)
- Content: many large binaries (textures/meshes/7z), typical file sizes KBs–GBs
Observed
| Commit size |
Result |
| ~610 GB (single seed commit) |
Working set climbed continuously (64 → 95 GB observed over ~25 min); killed before OOM. System commit charge 191 GB of 240 GB limit and rising. Server store had received 0 bytes at kill time. |
| ~80 GB (one directory) |
WS ~45 GB and climbing, modified page list 48 GB, Available MBytes = 0 on the 128 GB machine; killed. |
| ≤12 GB batches (same content, 50 commits) |
Peak WS ~18 GB per commit, steady; completes normally. |
CPU is near idle during the growth phase; the process appears to be reading source files (network volume at ~1.5 Gbps) and accumulating fragments in private memory before/while uploading.
Notes
[file] direct_io = true in the workspace config.toml made no observable difference to commit-side behavior — reading the config reference, the [file] table seems to govern working-tree writes (checkout/sync), so there may be no commit-side knob today.stage --scan itself is fine at this scale (648k adds in ~46 s) — the growth is entirely in commit.- Happy to provide more measurements (this repo makes a convenient large-scale testbed).
Workaround
Split the initial seed into size-capped batches (we used ~12 GB per commit → ~18 GB peak WS). Fine as a workaround, but first contact with a large existing tree is exactly when a new user is least prepared for it.
Summary
Client commit memory grows roughly linearly with the size of the committed batch (~1.5x batch bytes peak). Seeding a large existing tree as one commit exhausts a 128 GB workstation; splitting the same content into small batches completes fine with bounded memory. Expected: commit streams fragments with backpressure, keeping memory roughly constant regardless of commit size.
Environment
lore 0.8.5+312, Windows 11 Pro for Workstations (Ryzen 9 7950X, 128 GB RAM)loreserver 0.8.5built from source (lore-server/Dockerfile, linux/amd64), Docker on a Synology DS1821+, local filesystem storesObserved
CPU is near idle during the growth phase; the process appears to be reading source files (network volume at ~1.5 Gbps) and accumulating fragments in private memory before/while uploading.
Notes
[file] direct_io = truein the workspaceconfig.tomlmade no observable difference to commit-side behavior — reading the config reference, the[file]table seems to govern working-tree writes (checkout/sync), so there may be no commit-side knob today.stage --scanitself is fine at this scale (648k adds in ~46 s) — the growth is entirely incommit.Workaround
Split the initial seed into size-capped batches (we used ~12 GB per commit → ~18 GB peak WS). Fine as a workaround, but first contact with a large existing tree is exactly when a new user is least prepared for it.