Repository navigation
Add model download cancellation and progress #111
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: develop
Are you sure you want to change the base?
Changes from all commits
5ce61ff
946fe01
8f29e68
4f3e24d
2fc0e30
425fff9
6340c6c
71f780a
78378cd
654dce6
c5ea671
174593e
baad93d
5936d2f
7a4d689
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -200,6 +200,23 @@ closed `EngineType` union and narrows external strings with `isEngineType()`. | |
| `engineManagerName()` / `engineTypeFromManagerName()` are the one place that | ||
| translation to and from the engine manager's own spelling lives. | ||
|
|
||
| ### Model downloads | ||
|
|
||
| A download is started with `engine:action{pull_model}`, or | ||
| `engine:remote-pull-model` for a pinned peer, and stopped with | ||
| `engine:cancel-pull` or `engine:remote-cancel-pull`. All of them name their | ||
| target in a `model` field: an engine can be downloading several models at once, | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "several models at once"? I thought it was one at a time in a queue. |
||
| and `engine:pull-progress` carries the same field so every frame lands on the | ||
| row it belongs to. | ||
|
|
||
| Cancellation is a request, not a result. The engine manager stops the transfer | ||
| and settles the partial files before answering, and the row clears when the | ||
| pull itself settles. A peer is given a much longer budget to answer than the | ||
| desktop waits, on purpose — cutting a peer off part-way through a cancel is | ||
| worse than waiting for it — so the desktop giving up first means the cancel is | ||
| still running, not that it failed. The row stays in Canceling and the cancel | ||
| can be issued again. | ||
|
|
||
| ### Engine settings | ||
|
|
||
| Server port, proxy port, and the engine arguments are one authoritative | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -78,7 +78,7 @@ They do not imply a WebSocket connection. Browser clients are not supported. | |
| - `getInitialState()` returns engine statuses, models, active progress, and available | ||
| updates. | ||
| - `toggle`, `install`, `uninstall`, and `update` manage engine lifecycle. | ||
| - `pullModel`, `loadModel`, `unloadModel`, `deleteModel`, and | ||
| - `pullModel`, `cancelModelPull`, `loadModel`, `unloadModel`, `deleteModel`, and | ||
| `setModelExpiry` manage models. | ||
| - `searchHub(engineType)` returns the curated model catalog for an engine. | ||
| - `onStateChanged`, `onProgress`, and `onProgressRemove` expose backend truth. | ||
|
|
@@ -112,6 +112,7 @@ environment assignments pass through without an engine-option catalog. | |
| - `uninstall`; | ||
| - `update`; | ||
| - `pullModel`; | ||
| - `cancelModelPull`; | ||
| - `loadModel`; | ||
| - `unloadModel`; | ||
| - `deleteModel`; | ||
|
|
@@ -121,6 +122,17 @@ Commands return no state. Renderer stores update from | |
| `engines:state-changed`, `engines:progress-changed`, and | ||
| `engines:progress-cleared`. | ||
|
|
||
| The model-bearing commands — `pullModel`, `cancelModelPull`, `loadModel`, | ||
| `unloadModel`, `deleteModel`, and `setModelExpiry` — carry the target in | ||
| `model`. It is what lets a node run several downloads at once and have each | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "several downloads at once" - same concern as architecture.md |
||
| one cancelled, and progress-tracked, on its own. | ||
|
|
||
| `cancelModelPull` reaches `engine:cancel-pull`, or `engine:remote-cancel-pull` | ||
| when `nodeId` names a peer. The download is not cancelled when the command | ||
| returns: the row moves to Canceling and clears when the pull itself settles. A | ||
| cancel whose request outlives its budget leaves the row in Canceling, because | ||
| the backend is still working on it, and stays available to issue again. | ||
|
|
||
| ### `pairApi.workloads` | ||
|
|
||
| - `getInitial()` returns active workloads. | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -121,6 +121,7 @@ reserved for inference clients. | |
| | `engine:ready` / `engine:state-changed` | Update engine facts and models | `engines:state-changed` | | ||
| | `engine:settings-changed` | Validate and republish the owning node's settings snapshot | `engines:settings-changed` | | ||
| | `engine:install-progress` / `engine:remote-progress` | Update operation progress | engine progress pushes | | ||
| | `engine:pull-progress` | Advance the optimistic pull entry the `model` field names | engine progress pushes | | ||
| | `errors:update` | Replace the error snapshot | `errors:update` | | ||
| | `cluster:invite-received` | Parse inbound invite | `cluster:invite-received` | | ||
| | `cluster:invite-canceled` / `cluster:invite-expired` | Prune the canceled or timed-out inbound invite from the authoritative set | `cluster:pending-invites-changed` | | ||
|
|
@@ -153,6 +154,26 @@ Local engine operations include install, start, stop, uninstall, update, port | |
| changes, and model actions. Remote cluster operations use the engine manager's | ||
| remote control surface where supported. | ||
|
|
||
| ### Cancelling a download | ||
|
|
||
| `engine:cancel-pull` stops a download on this node and | ||
| `engine:remote-cancel-pull` stops one on a pinned peer. Both name their target | ||
| with a `model` field, which is what distinguishes them from the engine-wide | ||
| commands: an engine may have several downloads in flight, and the frames on | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "several downloads in flight" - same concern as architecture.md although I think this is closer to reality. |
||
| `engine:pull-progress` carry the same field so each one lands on its own row. | ||
|
|
||
| Neither returns state. The backend answers only once the transfer has stopped | ||
| and its partial files are settled, so the reply is an acknowledgement and the | ||
| row clears from the pull's own settling, not from the cancel. | ||
|
|
||
| That answer can outlive the desktop's budget. A peer is served by the engine | ||
| manager's readiness client, whose response-header budget is far longer than | ||
| `MODULAR_CANCEL_PULL_TIMEOUT_MS`, because cutting a peer off mid-cancel is | ||
| worse than waiting for it. A timeout therefore means "still cancelling", not | ||
| "failed": the row stays in Canceling rather than dropping back to Downloading, | ||
| and the cancel can be issued again. Only an explicit rejection restores the | ||
| previous status and reports an error. | ||
|
|
||
| ### Engine settings | ||
|
|
||
| Ports and the engine arguments are one authoritative, revisioned record owned by | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -267,15 +267,126 @@ the engine simply shows as stopped and then running. | |
|
|
||
| A local `pull_model` streams live download progress: the engine-manager routes | ||
| `engine:action{pull_model}` through its streaming pull path and emits | ||
| `engine:pull-progress` (`{ engine, op, stage, percent, message }`) — the local | ||
| `engine:pull-progress` (`{ engine, model, op, stage, percent, message }`) — the local | ||
| counterpart of `engine:remote-progress`. Personal AI Router consumes it in | ||
| `applyLocalEngineProgress` (`modular-supervisor.ts` → `modular-state.ts`), | ||
| backfilling the dispatched model (the frame carries none) and advancing the | ||
| optimistic pull entry's percent in place, so a local pull shows "Pulling · N%" | ||
| keying the frame to its optimistic pull entry by model and advancing that | ||
| entry's percent in place, so a local pull shows "Pulling · N%" | ||
| to completion just like a remote pull. The awaited action response owns | ||
| completion (clearing the entry and refreshing the model list); a CLI-driven pull | ||
| (LM Studio) emits a single `pulling` marker and degrades to the indeterminate | ||
| spinner. | ||
| completion (clearing the entry and refreshing the model list). LM Studio's CLI | ||
| progress updates the same percentage display. A pull for an engine that already | ||
| has one running is reported as `stage:"queued"` until its turn. | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can you clarify that it joins onto an existing download? We don't download it twice, right? |
||
|
|
||
| The pull request itself stays pending for the whole download, so it is the | ||
| request's own response that reports the outcome — a completion or a | ||
| cancellation — while progress arrives out of band on the notification: | ||
|
|
||
| ```mermaid | ||
| sequenceDiagram | ||
| participant UI as Model Manager | ||
| participant Bridge as Electron bridge | ||
| participant Manager as Engine manager | ||
| participant Engine as Ollama or LM Studio | ||
|
|
||
| UI->>Bridge: Download model | ||
| Bridge->>UI: Show optimistic "Pulling…" row | ||
| Bridge->>Manager: Start pull (request remains pending) | ||
| Manager->>Manager: Record the partial files already on disk | ||
| Manager->>Engine: Begin download | ||
|
|
||
| loop While downloading | ||
| Engine-->>Manager: Download status | ||
| Manager-->>Bridge: Progress {engine, model, stage, percent} | ||
| Bridge-->>UI: Update the matching model row | ||
| end | ||
|
|
||
| alt Download completes | ||
| Engine-->>Manager: Success | ||
| Manager-->>Bridge: Original pull request completes | ||
| else User cancels | ||
| UI->>Bridge: Cancel download | ||
| Bridge->>UI: Show "Canceling…" | ||
| Bridge->>Manager: Cancel {engine, model} | ||
| Manager->>Engine: Stop transfer | ||
| Manager->>Manager: Remove the partial files this pull created | ||
| Manager-->>Bridge: Cancellation completes | ||
| Manager-->>Bridge: Original pull settles as canceled | ||
| end | ||
|
|
||
| Bridge->>Manager: Refresh model list | ||
| Manager-->>Bridge: Authoritative models | ||
| Bridge->>UI: Replace temporary download state | ||
| ``` | ||
|
|
||
| `engine:cancel-pull` and `engine:remote-cancel-pull` cancel the selected model | ||
| download. The UI keeps "Canceling…" visible until the pull settles, including | ||
| when the cancel RPC's own budget elapses first — the backend is still working, | ||
| so the row is not dropped back to "Downloading". Ollama closes the pull request | ||
| and removes partial blobs named by its progress digests; LM Studio receives | ||
| Ctrl+C and a negative answer to its background-download prompt, and after | ||
| acknowledgement the partial files of the model file it was fetching are | ||
| removed. That file is identified by the requested quantization when the request | ||
| pins one, and otherwise by the file names the CLI printed; an unpinned pull | ||
| whose output named no file removes nothing. Completed model files, other | ||
| quantizations, and unrelated downloads are retained, and a cancellation the CLI | ||
| never confirmed deletes nothing. Only a cancellation someone requested removes | ||
| files: a pull interrupted by app shutdown, a dropped remote connection, or the | ||
| action timeout leaves its partial data resumable. | ||
|
|
||
| Cleanup looks for partial files only in the default model directories: | ||
| `~/.lmstudio/models` for LM Studio, and for Ollama the `OLLAMA_MODELS` the | ||
| engine-manager launches it with (`~/.ollama/models` by default). A cancellation | ||
| on an install whose models live elsewhere — a relocated LM Studio models | ||
| directory, or an adopted Ollama daemon started with a different | ||
| `OLLAMA_MODELS` — stops the download but leaves its partial files for the | ||
| engine to resume or remove. | ||
|
|
||
| Both engines funnel every client on the machine into one cache, so naming a file | ||
| is not the same as owning it — Ollama blobs are content-addressed, and the LM | ||
| Studio app downloads into the same repository directory. Each pull therefore | ||
| records the partial files already present before it starts, and cancelling it | ||
| considers only the ones that appeared afterwards. A file that is still growing | ||
| once the transfer has stopped is shared with another client even by that | ||
| measure, so it survives as well: | ||
|
|
||
| ```mermaid | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This diagram might want to mention if the existing request is already in progress. From trackedPull "A second request for a download already in flight joins it" |
||
| flowchart TD | ||
| Request["Model pull requested"] --> Tracked["Track by engine + model"] | ||
| Tracked --> Busy{"Another PAIR pull active<br/>for this engine?"} | ||
|
|
||
| Busy -- No --> Snapshot["Record the partial files<br/>already on disk"] | ||
| Busy -- Yes --> Queued["Show as queued"] | ||
|
|
||
| Queued --> QueueCancel{"Canceled while queued?"} | ||
| QueueCancel -- Yes --> NeverStarted["Remove operation<br/>without starting download"] | ||
| QueueCancel -- No --> Wait["Wait for active pull to settle"] | ||
| Wait --> Snapshot | ||
|
|
||
| Snapshot --> Active["Start download"] | ||
| Active --> Cancel{"Cancellation requested?"} | ||
| Cancel -- No --> Complete["Download completes normally"] | ||
| Cancel -- Yes --> Engine{"Which engine?"} | ||
|
|
||
| Engine -- Ollama --> OllamaStop["Close HTTP pull request"] | ||
| OllamaStop --> OllamaCandidates["Select partial blobs named<br/>by this pull's digests"] | ||
|
|
||
| Engine -- LM Studio --> LMSStop["Send Ctrl+C"] | ||
| LMSStop --> LMSPrompt["Answer no to background download"] | ||
| LMSPrompt --> LMSCandidates[".part files of the pinned<br/>quantization or a file the CLI named"] | ||
|
|
||
| OllamaCandidates --> Owned{"Absent from<br/>the snapshot?"} | ||
| LMSCandidates --> Owned | ||
| Owned -- No --> Preserve["Preserve completed,<br/>shared, and unrelated files"] | ||
| Owned -- Yes --> Quiet{"Held still across<br/>repeated checks?"} | ||
| Quiet -- No --> Preserve | ||
| Quiet -- Yes --> Delete["Remove the partial file"] | ||
|
|
||
| Delete --> Settle["Settle pull and refresh models"] | ||
| Preserve --> Settle | ||
| Complete --> Settle | ||
| NeverStarted --> Settle | ||
| Settle --> Next["Allow next queued pull to start"] | ||
| ``` | ||
|
|
||
| `engine:models` (and the `em` `GET /v1/models` surface) returns the flat model | ||
| union, the per-engine breakdown (`modelsByEngine`), and the per-engine set of | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LMStudio has "model", but IIUC, Ollama has
{name: model}