Skip to content

Promote fine-tuned LokalBot models to defaults with upgrade nudge - #561

Open
stevyhacker wants to merge 1 commit into
masterfrom
feat/gemma4-qwen3-asr-lokalbot-models
Open

stevyhacker wants to merge 1 commit into
masterfrom
feat/gemma4-qwen3-asr-lokalbot-models

Conversation

@stevyhacker

@stevyhacker stevyhacker commented Nov 21, 2026 •

Copy link
Copy Markdown
Owner

Summary

Rebased cleanly on master (d92e1401). Promotes all four Apache-2.0 LokalBot fine-tuned models to the new built-in defaults and adds a dismissible upgrade nudge for existing users who are still on the previous stock defaults. This supersedes #557 (which added Qwen3.5-4B-LokalBot-v2 as an opt-in entry).

All four models were trained exclusively on open-license public and synthetic data (Apache-2.0, ODC-By, CC0, CC-BY-4.0, zero private user data, zero CC-BY-SA, disjoint from eval/holdout splits).

New Default Models (Immutable Hugging Face Pins)

  1. Think (macOS + Linux/Windows): Qwen3.5 4B (LokalBot v3) (qwen3.5-4b-lokalbot-v3)
    • Repo: stevyhacker/Qwen3.5-4B-LokalBot-GGUF @ 60f704305746c4390c3b207c156191c761cf66db
    • File: Qwen3.5-4B-LokalBot-v3-Q4_K_M.gguf (2,741,241,984 B, SHA-256 ff8e5921a49c1a4d9cf8a22759fb83deac4b7670d95d659e268e565e7af1ec03)
    • Excluded from Autocomplete pickers; stock Qwen3.5 4B (qwen3.5-4b) stays available in the catalog.
  2. Autocomplete (macOS + Linux/Windows): Gemma 4 E2B (LokalBot v4) (gemma4-e2b-lokalbot-v4)
    • Repo: stevyhacker/Gemma-4-E2B-LokalBot-GGUF @ e089648312aee558514a13dbbd4efb19a0b3eb25
    • File: Gemma-4-E2B-LokalBot-v4-i1-Q6_K.gguf (3,845,323,008 B, SHA-256 ea8c01dd28abaae0a6664eb9c1324f6bf6cd0ffef99dace496510f7cceb40581)
    • Excluded from Think pickers; stock Gemma 4 E2B Base (Autocomplete) (gemma4-e2b-base-q6) stays available in the catalog.
  3. Transcribe 1.7B (macOS QwenLlamaASR.accuracy): Qwen3-ASR-1.7B-LokalBot-v3
    • Repo: stevyhacker/Qwen3-ASR-1.7B-LokalBot-GGUF @ 33ca434ce773d56914f609cec80e843d085b5645
    • Decoder: Qwen3-ASR-1.7B-LokalBot-v3-Q8_0.gguf (2,165,035,040 B, SHA-256 a3611b68d5065e3984d0cf8a93adaf7cfaf5f4a09b33da8d68914a7ae85aee19)
    • Projector: mmproj-Qwen3-ASR-1.7B-LokalBot-v3-Q8_0.gguf (355,709,376 B, SHA-256 0eb673620a73c412a0de2bb31a427ac5367f686e14db0922944023e44356c785)
  4. Transcribe 0.6B (macOS QwenLlamaASR.compact): Qwen3-ASR-0.6B-LokalBot-v1
    • Repo: stevyhacker/Qwen3-ASR-0.6B-LokalBot-GGUF @ 4f9e435248343b208223ce33fa1a8b0b2f2e6bc6
    • Decoder: Qwen3-ASR-0.6B-LokalBot-v1-Q8_0.gguf (804,749,344 B, SHA-256 c7493d2b337bb351f1c562337e7c9fdfef0195a26d4e91cfae805a6416066c3c)
    • Projector: mmproj-Qwen3-ASR-0.6B-Q8_0.gguf (214,392,480 B, SHA-256 41a342b5e4c514e968cb756de6cd1b7be39eff43c44c57a2ef5fc6522e36603d, bit-identical to stock ggml-org)

Upgrade Nudge & Non-Breaking Transition for Existing Users

  • Existing users who have already saved qwen3.5-4b, gemma4-e2b-base-q6, or stock Qwen3-ASR GGUF files keep working without surprise downloads or forced switches.
  • On macOS, QwenLlamaASR falls back to verified legacy stock Qwen3-ASR GGUF weights on disk when the new files are not downloaded yet, so offline transcription continues working until the user triggers an upgrade. Upgrading downloads the new weights, restarts the ASR server if the projector changed, and removes superseded stock decoder/projector files (preserving the shared stock 0.6B mmproj).
  • A dismissible upgrade card appears on Today (suppressed on fresh installs, demo libraries, and when Today prompts are suppressed) and in Settings > Models whenever at least one active built-in role still points at a superseded stock default:
    • Think still set to qwen3.5-4b (built-in backend)
    • Autocomplete still set to gemma4-e2b-base-q6 (built-in backend, only when Autocomplete is enabled or stock gemma4-e2b-base-q6 is already on disk)
    • Transcribe set to qwen3-asr-1.7b or qwen3-asr-0.6b with legacy stock GGUFs on disk (macOS)
  • Users on custom models or remote servers never see the prompt. Clicking Not now stores per-role dismissal tokens in dismissedDefaultModelUpgradeKey (registered as an internal settings key on both macOS and Linux/Windows). Clicking Upgrade downloads and switches in one step.
  • Added localized strings across all 24 LokalBot/Resources/*.lproj/Localizable.strings files, synced to LinuxWindows/crates/lokalbot-i18n/src/catalogs.rs, and added Apache-2.0 provenance entries for all four repos in THIRD_PARTY_NOTICES.md.

Benchmark Summary (Apple M4 Max)

Track / Metric Stock Fine-Tune Delta
Think Qwen3.5-4B v3 Memory Retention (120) · EN / SR recall 80/120 · 28/52 / 0/13 118/120 · 52/52 / 12/13 +38/120
Think Qwen3.5-4B v3 SparkX26 · Multi-Step 7/12 (37/43) · True 9/12 (40/43) · True +2 cases / +3 fields
Think Qwen3.5-4B v3 Golden6 dec / rej · Private12 dec mtgs 3/6 / 3 · 2/8 7 / 0 · 8/8 0 rej · 8/8 mtgs
Think Qwen3.5-4B v3 Private12 rej / rep · quote miss · med s 40 / 19 · 8 · 33.1 s 6 / 5 · 0 · 19.2 s -85% rej · -42% time
Autocomplete Gemma 4 E2B v4 Engine (788) acc / prec / wrong 23.10% / 50.14% / 181 25.25% / 52.23% / 182 +2.15 / +2.09 pp
Autocomplete Gemma 4 E2B v4 Memory (26) · Visible (30) · Email prec 26/26 · 28/30 · 74.3% 26/26 · 30/30 · 80.0% 56/56 · +5.7 pp
Autocomplete Gemma 4 E2B v4 FH v1 (160) / SR-Latn · FH v2 (82) 75.6% / 53.3% · 43.9% 78.8% / 76.7% · 52.4% +3.2 / +23.4 / +8.5 pp
Autocomplete Gemma 4 E2B v4 Med / Typing p95 ms 20.6 / 95.6 ms 20.2 / 80.0 ms -0.4 / -15.6 ms
Transcribe Qwen3-ASR 1.7B v3 AMI-400 ihm / sdm (200 ea) 10.24% / 26.08% 9.79% / 24.41% -0.45 / -1.67 pp
Transcribe Qwen3-ASR 1.7B v3 Silence Clean (24) · Code-Switch (17) · hr (120) 8.3% · 37.12% · 24.60% 100.0% · 12.81% · 17.59% +91.7 / -24.31 / -7.01 pp
Transcribe Qwen3-ASR 0.6B v1 South Slavic (120) · OOD hr (25) · Code-Switch (17) 52.26% · 46.95% · 49.21% 30.57% · 24.56% · 20.72% -21.69 / -22.39 / -28.49 pp
Transcribe Qwen3-ASR 0.6B v1 AMI-400 ihm / sdm · Vocab Recall (15) 10.87% / 26.57% · 80% 10.72% / 26.23% · 100% -0.15 / -0.34 / +20.0 pp

Known Tradeoffs

  • Qwen3.5-4B-LokalBot-v3: Emits fewer Private12 action items than stock (54 vs 74, 35.7% vs 50.0% segment recall) in exchange for 6.7x fewer validator rejections (6 vs 40), 0 missing ownership quotes, 8/8 decision meetings (vs 2/8), and 42% lower median latency.
  • Gemma-4-E2B-LokalBot-v4: Uses a stock-calibrated operating point (182 wrong shown vs 181 stock) so memory and visible context stay 56/56; conversation precision dips from 48.6% to 42.9% and formatting precision from 61.5% to 57.1%.
  • Qwen3-ASR-1.7B-LokalBot-v3: Returns empty output on 2/200 very low-SNR AMI-400 SDM clips (stock 0/200); 5-language guard WER edges from 4.35% to 4.53%; requires its paired v3 mmproj.
  • Qwen3-ASR-0.6B-LokalBot-v1: 5-language guard WER edges from 6.64% to 6.77%; 1/200 empty AMI-400 SDM clip.

Validation

Local and remote non-UI checks completed on commit 13ae3537:

  • python3 Scripts/ci/comment-ratchet.py --check: passed (Comment lines: 0 (baseline 0)).
  • swiftlint lint --strict --quiet on all 19 touched Swift files: passed (0 warnings/errors).
  • macOS targeted non-UI xcodebuild test (ModelCatalogTests, QwenLlamaASRTests, ModelRoleNamingTests, ModelStorageCopyTests, ModelRolesTests, ModelSettingsTests, CotypingSettingsTests, AppSettingsTests, SettingRegistryTests, OnboardingFlowTests, TodayLandingTests): 194 executed, 0 failures, 1 env-gated skip.
  • Full macOS non-UI LokalBotTests run (5,231 tests): caught OnboardingFlowTests/testBeforeDownloadingEachModelHasItsSizeAndTheTotalCountsOnlyWhatIsChosen, fixed and re-verified (24/24 passing); the only other local failures were 4 macOS Vision OCR tests unrelated to model catalog changes.
  • Linux/Windows checks:
    • (cd LinuxWindows && cargo fmt --all -- --check): passed.
    • (cd LinuxWindows && python3 -m unittest discover -s scripts/tests): 9/9 localization parity tests passed.
    • Full Linux workspace on Ubuntu 24.04 (cargo clippy --workspace --all-targets --locked -- -D warnings && cargo test --workspace --locked --no-fail-fast): 0 clippy warnings, 2,762 passed, 0 failed, 28 ignored.
  • Hugging Face reachability and LFS integrity verified (HTTP 302 with exact x-linked-size and SHA-256 x-linked-etag) on all 6 pinned URLs across all 4 repos.
  • macOS UI tests (XCUITest and Day in the life) were not run locally per AGENTS.md and will run on hosted CI now that this PR is marked ready for review.

Changed test expectations

  • LokalBotTests/Engines/ModelCatalogTests.swift (testQualityCotypingModelIsPinnedAndOnlyOfferedForAutocomplete): gemma4-e2b-base-q6 (gemma-4-E2B.i1-Q6_K.gguf, 3,845,328,608 B) changed to gemma4-e2b-lokalbot-v4 (Gemma-4-E2B-LokalBot-v4-i1-Q6_K.gguf, 3,845,323,008 B). Reason: Gemma 4 E2B (LokalBot v4) is now the default autocomplete model; the test still verifies that stock gemma4-e2b-base-q6 stays in the catalog via previousQualityCotypingID.
  • LokalBotTests/Engines/ModelCatalogTests.swift (testFreshSettingsDefaultToQwen35FourBOnEveryMac): ModelCatalog.defaultSummarizationID "qwen3.5-4b" changed to "qwen3.5-4b-lokalbot-v3" (Qwen3.5-4B-LokalBot-v3-Q4_K_M.gguf, 2,741,241,984 B). Reason: Qwen3.5 4B (LokalBot v3) is now the default Think model on fresh installs.
  • LokalBotTests/Engines/ModelCatalogTests.swift (testDefaultPresetsHavePlainNamesAndPickActualModels): "Transcribe: Qwen3-ASR 1.7B, Think: Qwen3.5 4B, Autocomplete: Gemma 4 E2B Base (Autocomplete)" changed to "Transcribe: Qwen3-ASR 1.7B, Think: Qwen3.5 4B (LokalBot v3), Autocomplete: Gemma 4 E2B (LokalBot v4)". Reason: the default model stack preset uses the new Think and Autocomplete defaults.
  • LokalBotTests/Engines/QwenLlamaASRTests.swift (testPinnedFilesLiveUnderTheQwenCacheAndPointAtTheRevision): stock ggml-org Qwen3-ASR 1.7B/0.6B URLs and filenames changed to the pinned stevyhacker/Qwen3-ASR-1.7B-LokalBot-GGUF (v3 decoder and paired v3 mmproj) and stevyhacker/Qwen3-ASR-0.6B-LokalBot-GGUF (v1 decoder and stock mmproj). Reason: both Qwen3-ASR tiers now pin the fine-tuned weights; new tests also cover legacy offline fallback and superseded weight cleanup.
  • LokalBotTests/Cotyping/CotypingSettingsTests.swift (testLegacyCotypingFieldsDecodeWithoutErrorAndAreNotWrittenBack, testCorruptedValuesFallBackFieldByField): default cotypingBuiltInModelID "gemma4-e2b-base-q6" changed to "gemma4-e2b-lokalbot-v4". Reason: fresh and fallback settings now default to the fine-tuned autocomplete model.
  • LokalBotTests/Models/Settings/AppSettingsTests.swift (testThinkModelDisplayNameFollowsSelectedBackendNotLeftoverLocalID): "Qwen3.5 4B" changed to "Qwen3.5 4B (LokalBot v3)". Reason: ModelCatalog.defaultSummarizationID now resolves to the v3 fine-tune.
  • LokalBotTests/Models/Onboarding/OnboardingFlowTests.swift (testBeforeDownloadingEachModelHasItsSizeAndTheTotalCountsOnlyWhatIsChosen): ["Qwen3-ASR 1.7B", "Qwen3.5 4B", "Gemma 4 E2B Base (Autocomplete)"] changed to ["Qwen3-ASR 1.7B", "Qwen3.5 4B (LokalBot v3)", "Gemma 4 E2B (LokalBot v4)"]. Reason: onboarding displays the active default Think and Autocomplete model names.
  • LinuxWindows/crates/lokalbot-core/src/settings.rs (tests::defaults_round_trip): "builtInModelID":"qwen3.5-4b" changed to "builtInModelID":"qwen3.5-4b-lokalbot-v3". Reason: Settings::default().built_in_model_id now defaults to Qwen3.5 4B (LokalBot v3).
  • LinuxWindows/crates/lokalbot-ai/src/catalog.rs (tests::autocomplete_defaults_to_gemma_e2b_as_on_macos): QUALITY_AUTOCOMPLETE_ID "gemma4-e2b-base-q6" changed to "gemma4-e2b-lokalbot-v4". Reason: matches the new default autocomplete model on both platforms.
  • LinuxWindows/crates/lokalbot-ai/src/think.rs (think::tests::built_in_client_names_the_model_on_this_pc): "Qwen3.5 4B · on this PC" changed to "Qwen3.5 4B (LokalBot v3) · on this PC". Reason: ThinkClient::new with Settings::default() now resolves to Qwen3.5 4B (LokalBot v3).
  • LinuxWindows/crates/lokalbot-agent/src/endpoint.rs (endpoint::tests::built_in_resolves_to_its_model_id): "qwen3.5-4b" changed to "qwen3.5-4b-lokalbot-v3". Reason: default agent endpoint uses Settings::default().built_in_model_id.
  • LinuxWindows/crates/lokalbot-engine/src/autocomplete.rs (tests::records_daily_stats_per_surface_and_model): Some("gemma4-e2b-base-q6") changed to Some("gemma4-e2b-lokalbot-v4"). Reason: autocomplete stats record the active default model ID from Settings::default().

Linked issues

Supersedes #557.

Risk / rollout notes

  • Settings compatibility: dismissedDefaultModelUpgradeKey defaults to "" on decode when missing from older settings files, and is registered in SettingRegistry.internalKeys (macOS) and INTERNAL_KEYS (Linux/Windows).
  • Existing users: No silent forced switches or unexpected downloads. Stock Think and Autocomplete models stay in the catalog and keep running if already configured; legacy Qwen3-ASR files in Qwen3-ASR-1.7B-Q8_0 / Qwen3-ASR-0.6B-Q8_0 continue working offline until the user accepts the upgrade prompt.
  • Linux/Windows speech: Speech on Linux/Windows still uses sherpa-onnx and is unchanged; the upgrade nudge there covers Think and Autocomplete.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
lokalbot 13ae353 Commit Preview URL

Branch Preview URL
Nov 22 2026, 06:00 AM

@stevyhacker
stevyhacker marked this pull request as ready for review November 21, 2026 06:08
@stevyhacker
stevyhacker marked this pull request as draft November 22, 2026 05:29
@stevyhacker
stevyhacker force-pushed the feat/gemma4-qwen3-asr-lokalbot-models branch from 253fb00 to 13ae353 Compare November 22, 2026 05:59
@stevyhacker stevyhacker changed the title Add Gemma 4 E2B (LokalBot v3) and update Qwen3-ASR 1.7B v3 / 0.6B v1 pins Promote fine-tuned LokalBot models to defaults with upgrade nudge Nov 22, 2026
@stevyhacker
stevyhacker marked this pull request as ready for review November 22, 2026 06:01

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant