Add local-only mode running Whisper on-device - #1
Open
PrestigePvP wants to merge 1 commit into
Open
PrestigePvP wants to merge 1 commit into
PrestigePvP wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes
provider.mode = "local"mean this Mac instead of a LAN server. Whisper runs in-process viawhisper-rs(whisper.cpp + Metal), so dictation works fully offline and audio never leaves the machine.Cloud mode is unchanged. The LAN-server path is removed from the app;
server/stays in the repo but is no longer referenced.Changes
local_engine.rs— drains PCM chunks, gates silence/short clips, s16le→f32, whisper inference inspawn_blockingwrapped incatch_unwind, filters non-speech markers, optional Ollama formatting. A formatter outage falls back to raw text rather than losing the transcript.models.rs— model catalog with pinned sizes + SHA-256, download from Hugging Face via.partial→ streaming hash verify → atomic rename. Progress events throttled to ~10/sec with the final byte always emitted.lib.rs— model kept warm inAppState, loaded on a background thread; dispatch atrun_pipelinebranches local vs cloud. A generation counter makes superseded loads discard themselves.config.rs—LocalConfigswapped fromserver_url/tokento on-device fields, all#[serde(default)].Measured on an M1 Pro
small.en, 11s clip: 459 ms inference (24x realtime), 2.1x faster than CPU. Model load 349 ms, which is why it's kept warm.Notes for review
config::loadtreats a deserialize failure as "try the next path" and writes a fresh default if all fail — so a required new field would silently wipe existing cloud credentials. Every new field is#[serde(default)], with tests covering the pre-change config shape.tiny.enreturns unpunctuated text, which is whysmall.enis the default and whyformatter: "none"is viable at all. The picker labels the smaller models accordingly.bundle.macOS.minimumSystemVersionandMACOSX_DEPLOYMENT_TARGET(withforce = true) in.cargo/config.toml. ggml usesstd::filesystem::path, which needs 10.15+; without both,cargo buildsucceeds whilecargo tauri buildfails.Testing
Two headless flags were added so local mode can be exercised without a mic or Accessibility permission:
Verified:
.apploads the model with Metal (the failure mode that only appears in a bundle)Not yet verified: hotkey registration and auto-paste (both need Accessibility permission and are unchanged by this PR), and the in-app downloader end-to-end (the model was copied in rather than fetched through the UI).