Skip to content

Add local-only mode running Whisper on-device - #1

Open
PrestigePvP wants to merge 1 commit into
mainfrom
local-only-whisper
Open

PrestigePvP wants to merge 1 commit into
mainfrom
local-only-whisper

Conversation

@PrestigePvP

Copy link
Copy Markdown
Owner

Makes provider.mode = "local" mean this Mac instead of a LAN server. Whisper runs in-process via whisper-rs (whisper.cpp + Metal), so dictation works fully offline and audio never leaves the machine.

Cloud mode is unchanged. The LAN-server path is removed from the app; server/ stays in the repo but is no longer referenced.

Changes

  • local_engine.rs — drains PCM chunks, gates silence/short clips, s16le→f32, whisper inference in spawn_blocking wrapped in catch_unwind, filters non-speech markers, optional Ollama formatting. A formatter outage falls back to raw text rather than losing the transcript.
  • models.rs — model catalog with pinned sizes + SHA-256, download from Hugging Face via .partial → streaming hash verify → atomic rename. Progress events throttled to ~10/sec with the final byte always emitted.
  • lib.rs — model kept warm in AppState, loaded on a background thread; dispatch at run_pipeline branches local vs cloud. A generation counter makes superseded loads discard themselves.
  • config.rsLocalConfig swapped from server_url/token to on-device fields, all #[serde(default)].
  • Settings UI — model picker with download progress and status badge; formatter selector with conditional Ollama fields.

Measured on an M1 Pro

small.en, 11s clip: 459 ms inference (24x realtime), 2.1x faster than CPU. Model load 349 ms, which is why it's kept warm.

Notes for review

  • Config migration is the risky part. config::load treats a deserialize failure as "try the next path" and writes a fresh default if all fail — so a required new field would silently wipe existing cloud credentials. Every new field is #[serde(default)], with tests covering the pre-change config shape.
  • tiny.en returns unpunctuated text, which is why small.en is the default and why formatter: "none" is viable at all. The picker labels the smaller models accordingly.
  • First-ever Metal shader init costs ~8.4s (0.007s thereafter, OS-cached). Preload is off the main thread — inline it would freeze launch for ~9s on first run.
  • Two build settings are load-bearing and must stay in sync: bundle.macOS.minimumSystemVersion and MACOSX_DEPLOYMENT_TARGET (with force = true) in .cargo/config.toml. ggml uses std::filesystem::path, which needs 10.15+; without both, cargo build succeeds while cargo tauri build fails.

Testing

Two headless flags were added so local mode can be exercised without a mic or Accessibility permission:

voicebox --transcribe-file audio.wav   # WAV -> transcript
voicebox --record-test 9               # record N seconds from mic -> transcript

Verified:

  • WAV → transcript, 470–543 ms
  • Mic → transcript via acoustic loopback (audio played through speakers, captured by the built-in mic, transcribed verbatim) — covers cpal device selection, 48k→16k resampling, and chunking
  • Silence and near-silence → empty, gated in 7–8 ms without reaching the model
  • Bundled .app loads the model with Metal (the failure mode that only appears in a bundle)
  • Pre-change config parses with credentials intact
  • 25 tests, clippy, and tsc clean

Not yet verified: hotkey registration and auto-paste (both need Accessibility permission and are unchanged by this PR), and the in-app downloader end-to-end (the model was copied in rather than fetched through the UI).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant