Smart AI STT Auto Scripture & LLM APIs - #3579
Conversation
3083d2f to
f187e0c
Compare
|
Pushed a round of updates from live field testing at a conference, plus a fully-local AI option:
141 unit tests passing. The PR description's update section has the full list. |
3aca772 to
0e03d7d
Compare
|
Left a longer note on #3547 about complementary fully local streaming work I have been testing (no API keys / no LLM for the core path). Happy to collaborate or wait for maintainer preference so we do not step on this review. |
|
@AlloDel your driver is merged and wired up — engine selector in settings, resumable model download, packaging config. Thank you, the VAD choice and the Electron notes were spot on. One change to your driver, after live testing. The decoder deafness you documented goes deeper than stream reuse: any chunked feeding (fresh, reused, or persistent stream) intermittently decodes short utterances like "next verse" to nothing. The same samples in one batch acceptWaveform on a fresh stream decode correctly every time. So the driver now buffers each utterance (1s pre-roll) and batch-decodes when the VAD closes, waiting 0.5s of real tail audio first so a soft last word isn't cut mid-word. VAD retuned: threshold 0.3, min-silence 0.8s, min-speech 0.15s. If your live-testing notes contradict any of this, I'd like to know. What I'd like you to review: downloading the sherpa runtime on demand — finishing your zero-setup goal. Right now the npm natives ship in the installer (~16 MB/platform) for everyone. The plan, mirroring our ffmpeg/whisper downloaders: Exact-pin sherpa-onnx-node@1.13.4; exclude the platform packages from electron-builder, keep the JS wrapper bundled Does the sibling-layout require() hold across sherpa-onnx-node versions you've used? |
5c74a02 to
5204dc3
Compare
|
I will review this soon. But just a couple things for now I would like changed:
|
|
Alright @vassbo |
Speakers just say "next chapter" - the grammar required an order like
"go to the next chapter", so the plain form never matched and the
command appeared broken in live use.
Without an imperative the phrase must END the utterance, which is what
separates an instruction from narration that contains the same words:
"in the next verse paul says something amazing" keeps talking, so it
still never fires, and a preposition right before the phrase ("we will
see that in the next chapter") is still read as narration. Applies to
relative movement, translation cycling and named translations. Verse
and chapter jumps stay imperative-only: a bare "verse 5" is already
resolved against the live passage by detection, and a bare "chapter 4"
is usually the tail of a spoken reference.
… word The VAD's confidence decays before a word actually ends, and audio was only fed to the recognizer while it flagged speech - the tail of an utterance's last word arrived after the flag dropped and went nowhere, so "next verse" came out as "next". While an utterance is open, keep feeding it through the dip; the VAD still decides where the utterance closes, so this only adds trailing audio.
Live testing kept losing short utterances ("next verse" transcribed as
"Next", or nothing at all) even after the audio was proven to reach the
recognizer. Measured against every feed pattern the binding supports,
the NeMo transducer decodes reliably in exactly one shape: a fresh
stream given the whole utterance in a single acceptWaveform call. Small
incremental chunks - on a fresh stream, a reused stream or one
persistent session stream - intermittently decode short utterances to
nothing, which is the deafness the original driver worked around but
could not eliminate.
The driver now buffers the utterance's audio (starting with 1s of
pre-roll so the encoder has lead-in before the first word) and batch
decodes it when the VAD closes. The close waits for half a second of
real tail audio first: a soft last word makes the VAD's silence
countdown run during the word, so the close can land mid-word - and if
speech resumes in that window it was a pause, so the utterance simply
continues. VAD tuning (threshold 0.5 -> 0.3, min silence 0.6s -> 0.8s,
min speech 0.25s -> 0.15s) keeps quiet word endings counted as speech
in the first place.
Validated against multi-utterance session audio: every utterance now
produces a segment, split and soft-tailed commands included.
The streaming engine emits one segment per utterance, so a breath in
the middle of a command splits it ("next" / "verse") and per-segment
matching can never see the whole phrase. CommandStream joins the
segments of the last few seconds and runs detection over the joined
tail; a command only fires when its matched phrase reaches into the
newest segment, so text that already had its chance never re-fires
from a later join, while genuinely repeating the command does.
Also from review: the start path now rejects the streaming engine
cleanly when the native addon is unavailable ("nemotron_unsupported"
instead of an unsanitized MODULE_NOT_FOUND require stack - the model
being downloaded does not imply the addon loads), and a start failure
message is sanitized like every other surfaced error.
The unsupported warning replaced the whole engine section, so a 660 MB model downloaded before the addon stopped loading had no way to be reclaimed from the UI.
The model URLs pointed at the repo's mutable "main" ref with only a size check on arrival, so what landed was whatever the CDN served. The URLs are now pinned to a specific revision and every file is verified against its SHA-256 (the LFS checksums Hugging Face publishes for that revision, plus the VAD gate's hash) before it counts as downloaded - a corrupt, truncated or substituted file is deleted and reported instead of landing. Files left by an earlier interrupted download are verified the same way before being skipped. Mid-transfer drops still retry with a ranged request inside one download call; a new call starts the file clean.
The feature's 108 strings were spread through the existing "settings" and
"scripture" sections, which every translator and most feature branches
touch. They now live under one "ai_scripture" parent at the bottom of
en.json, and each key drops the redundant "ai_" prefix the parent already
carries ("ai_whisper_model_tiny" -> "whisper_model_tiny").
settings.ai_scripture deliberately stays where it is: the settings tab
label is resolved from the tab id as settings.{tabId}, so moving it would
leave the tab and the profile permission list unnamed.
Two strings collapsed onto the same name once the prefix was dropped, so
the long notice shown when the feature is off is now "privacy_notice",
leaving "privacy" to the section heading that owns the _local/_llm/_keys
strings under it.
The feature was spread across src/electron/aiScripture, src/frontend/audio and two components sitting in unrelated component folders. It now lives in one place per process: src/electron/ai/ (was electron/aiScripture) src/frontend/ai/ (was the aiScripture files in audio/) src/frontend/components/ai/ (was settings/tabs + drawer/bible) The settings tab component is renamed AiScriptureSettings.svelte, since sitting beside AiScripturePanel.svelte it needed to say which one it is. The shared types stay in src/types/AiScripture.ts, which is where every type crossing the electron/renderer boundary lives. Both components moved a directory level up, so their relative imports lost one "../"; the panel's same-folder import of the bible helper is now an explicit path, and the whisper/audio helper imports follow their modules.
5204dc3 to
082dce5
Compare
|
Hi @vassbo |
- Changed data store - AI settings - Moved types
|
I made some changes.
|
|
I'll leave the bubble work for now. |
# Conflicts: # config/building/electron-builder.yaml # src/frontend/utils/popup.ts
|
@josephomills Do you have any more plans on this specific PR, or what's the status? |
|
Okay sounds great. Ultimately, my plan is basically to get STT, detections and quote-matching as accurate and fast as possible. I'm coming from a background of scripture-heavy sermons, no notes before service, switching between translations fluently and we already have a skilled projection team (not to boast, lol). We've tried other dedicated AI scripture tools which haven't worked (in terms of accuracy, speed, limit on available translations & other features). So if we can get this to work well, then I can say confidently that it'll work fine for all types of congregations. And I'm personally not for building another AI scripture software, I'd rather contribute to have the feature on FreeShow cos it does much more |
|
I think cross-translation quote-matching or switching translations via voice would be my next in line. Next will be handling paraphrasing. |
- Dynamic model loading - Renamed some files - Error messages - UI cleanup
|
I think the whisper path check has a bug. |
I know, it has not worked for a while. |
|
For the cross-translation quote-matching, if any user has many translations, the app will freeze while the bibles are being indexed. That's why I had run it on a worker initially. Open to indexing in a web worker? or should I index in sequence one at a time in the background (opened translation first then favourites then the rest). Do you have any idea the average number of translations users have? people in my circles will have 15+ |
|
I'm pretty certain that worker you had only worked in the dev build, and would not work in a built version. I added searching in multiple scriptures.
I guess most churches have just one or two translations that they commonly use. |
vassbo
changed the title
|
Merged! Thanks for helping me with this. :) It's at least working as a basic level, but now you can more easily experiment with the system and improve as much as you want on a seperate branch, then send a PR at any time. Got the floating color style improved as well: |
Great!
Nice! Will check it out
👍🏽
Exciting!
Yhup yhup. Thank you! Beta release coming soon? |
|
Thanks for all the work on this. Can't wait to try it out! |
* Check textFit when detecting items that need auto size (#3660) * Planning Center item type categories * Updated Hungarian language * Updated languages * Fixed action output update custom activations triggering too early #3667 * Wider title search spelling mistake range #3586 * Update * Fixed scripture split in two not updating properly #3573 * Fixed cutting scripture show text not updating properly #3574 * Fixed removing scripture show verse numbers * Scripture show cut/paste fix * Fix * Canva download cache resets if updated * Fixed active scripture book not visible in grid mode * Updated Chinese language * Stripped bundled dist file of ~10MB * Highlight to be deleted slides from context menu * Added audio settings to profiles #3671 * Fixed camera cropping * Fix minor keyboard shortcut issues (#3678) * Updated language * Item cutout shape #3669 * Custom vertial alignment with Cutout shape * Updated Polish language * Project section notes dynamic value #3675 * Tweak * Video duration tweak * Fix PDF text export layout for multi-page songs (#3684) * Improved text parsing * PCO Item Details import * Open show action sets the correct project index - Fixed clear actions triggering "after" output update when run from "Start show" action * Fixed FFmpeg not working on some macOS #3676 * Don't delete cloud folder before backup * Rendering overhaul: GPU-native output capture for multi-output / 4K performance (#3681) * Cleanup * Fixed PDF randomly not outputting #3686 * Sync B1 plan item assignees into section notes (#3675) (#3685) * Tweak * Fixed fade out pausing videos with no audio * Per-item Text edit #3617 * Tweak * PPT tweaks * Profiles settings tabs are now linked to global tabs * Updated language * Fixed style overwrite font size issue #3694 * Fixed Vimeo URL parse issue * Fixing start_scripture API verse selection issue #3700 * Fixed audio sometimes not going through Main channel #3688 * Audio rounting should not be synced * Custom fonts manager #3650 * Jesus words stay red across bracketed phrases (#3709) * Scripture undertitles wrap on the output instead of being clipped (#3708) * Bracketed text keeps its template style inside red-letter verses (#3712) * Added OMT Support (#3693) * Cleanup OMT * Set group template the moment the group is added * Fixed multi item move * Better shift move * Scripture templates can style undertitles with {scripture_undertitle} (#3718) * Revert segment * Fixed incorrect stage resolution * Fixed YouTube video not muted in stage output #3717 * Tweaks * Per output encoder option * Settings UI Tweak * Fixed new stage output not focused * Fixed main window saved position issues #3719 * Fixed chord auto break #3670 * Project collapsed mode #3703 * Tweak * Smart AI STT Auto Scripture & LLM APIs (#3579) * Gradient button tweak * Auto update STT when download finishes * Cleanup * Increased suggestion log limit * Don't include unneded files in build * Auto output position tweak * Scripture helper location update * Fixed overlays preview not paused * Longer camera preview load delay * Fix * Fixed PowerPoint custom crop format #3720 * Tweaks * Version update --------- Co-authored-by: Joseph Mills <96480787+josephomills@users.noreply.github.com> Co-authored-by: hansli112 <31886587+hansli112@users.noreply.github.com> Co-authored-by: Pineapple <34526243+Pineapple-Pineapple@users.noreply.github.com> Co-authored-by: David Myers <david.d.myers@gmail.com> Co-authored-by: Jeremy Zongker <jeremy@zongker.net>
|
Installed the beta and it works fantastically on my home laptop. I'm running it live this weekend during Sunday school with a Ghanaian congregation here in Italy, off a mixer feed, and I'll report precision and recall on detected references, accented English is a case I haven't seen covered here. One thing from setup: the only way to stop the listener mid-service is the settings tab. During a service the operator needs that in one click, a toggle and a listening indicator in the main interface would help. Is there already a status element in the drawer panel that could be promoted, or did that go in the rewrite? |
7 participants
What
AI auto scripture (opt-in, off by default): FreeShow listens to the sermon, transcribes locally with whisper.cpp, detects scripture references — both spoken ("John chapter 3 verse 16", instant local matching) and quoted ("for God so loved the world…", via a bring-your-own-key LLM) — and either projects them automatically or offers one-click suggestion cards in the scripture drawer.
https://github.com/ChurchApps/FreeShow — demo video to follow.
Features
Guarantees (please review these first)
safeStoragefollow-upPlatform matrix
brew install whisper-cppdetection or custom pathTesting
Happy to split, adjust scope, or rework anything — I kept every touched upstream file minimal and matched existing patterns (LTC worklet, ffmpegManager, ACCESS store, Material inputs, i18n) throughout.
Updates since opening
Test suite is now at 141.