Skip to content

Turn-by-turn live text on this Mac, with the measurements behind it - #479

Merged
stevyhacker merged 2 commits into
masterfrom
claude/turn-by-turn-live-text
Nov 11, 2026
Merged

stevyhacker merged 2 commits into
masterfrom
claude/turn-by-turn-live-text

Conversation

@stevyhacker

Copy link
Copy Markdown
Owner

Summary

Fast live text was the last feature that needed Meta's API. This adds the same thing with the Transcribe model on this Mac: Turn-by-turn live text (Settings → Meetings → Processing, off by default). The live preview cuts each track at pauses and transcribes one turn at a time, so a turn's text follows its last word by about half a second, where the preview's 4 to 12 second windows take about two seconds and up to five. Live Radar then reads after each turn. Muse's fast live text is unchanged and keeps precedence when it is on.

It was measured before it was built (report). The sample call as a microphone track and a call track, on an M4 Max, from a turn's last word to its text:

Model Recording Turns: median / worst Windows today: median / worst Word errors: turns / windows
Parakeet TDT 0.6B v3 English 0.50 / 0.81 s 1.86 / 3.83 s 3.4% / 8.7%
Qwen3-ASR 1.7B (default) English 0.64 / 0.86 s 1.89 / 4.29 s 2.9% / 8.5%
Whisper large-v3 turbo English 1.32 / 1.52 s 3.22 / 5.01 s 5.8% / 7.6%
Qwen3-ASR 1.7B Serbian 0.71 / 0.94 s 2.23 / 4.15 s 22.3% / 31.8%
Whisper large-v3 turbo Serbian 1.37 / 1.71 s 2.44 / 4.57 s 15.6% / 24.2%

Muse Voice Transcribe measured 0.74 s (median) on other audio. Turns took no more CPU time than windows.

How it works:

  • LiveTurnCutter reads speech detection's verdict on each 256 ms chunk. A turn closes after two quiet chunks, after one once it is 4 s long (this catches 23 and 24 of 25 speaker changes when two voices share a track, against 18 and 15), or at 15 s at a quiet point.
  • LiveMeetingTranscriber.runTurns hears both live files every 100 ms and transcribes closed turns one at a time. A failed turn is retried, and a turn waiting for the model survives a pause of the preview.
  • While the model answers within 0.25 s and runs on this Mac, the turn still being spoken is transcribed again every 2 s and shown as a partial line. A transcription server hears each turn once, when it is finished.

The report also covers a streaming recognizer the app already links (words 0.34 s behind the speaker and 4.7% word error in English, but 43.9% on the Serbian recording), and what the sherpa-onnx and Apple routes would take.

Validation

Mac, on this branch's files (identical to the tree tested, apart from the merge commit of #478):

xcodegen generate
xcodebuild -project LokalBot.xcodeproj -scheme LokalBot -configuration Debug \
  -destination 'platform=macOS' CODE_SIGNING_ALLOWED=NO -only-testing:LokalBotTests build-for-testing
# ** TEST BUILD SUCCEEDED **
xcodebuild test-without-building -xctestrun … -destination 'platform=macOS,arch=arm64' \
  -only-testing:LokalBotTests/{LiveTurnTextTests,LiveTurnCutterTests,LiveTurnLatencyBenchmarkTests,LiveMeetingTranscriberTests,MuseLiveTextTests,MeetingRadarTests,SettingRegistryTests,AppLanguageTests,LiveAnswersTests,RadarScreenAnswersTests}
# Executed 109 tests, with 3 tests skipped and 0 failures (the 3 are the opt-in measurements)
swiftlint lint --strict --quiet                                               # exit 0
python3 -m unittest discover -s Scripts/tests                                 # Ran 222 tests, OK
uv run --no-project python -m unittest discover -s Scripts/tests -p test_localization.py   # Ran 6 tests, OK
(cd LinuxWindows && python3 scripts/sync-localizations.py --check)            # exit 0
(cd LinuxWindows && python3 -m unittest discover -s scripts/tests)            # Ran 8 tests, OK
  • Seen failing. With deliversTurns put back, the server guard removed and a turn taken off the queue before it is transcribed, 5 of the 12 LiveTurnTextTests fail (the ones for those three things) and pass again with the code restored. The span-read test failed first on its own: one read of the live file returned 15,552 of 16,000 frames, which is why span reads now loop.
  • End to end. LiveTurnLatencyBenchmarkTests/testPreviewReplayInRealTime wrote 90 s of the English call to disk at recording speed and ran LiveMeetingTranscriber on it with the real speech detection and model: text 0.62 s (median) and 0.94 s (worst) after a speaker's last word with Parakeet v3, 0.70 and 0.96 s with Qwen3-ASR 1.7B, with the turn in progress shown 31 times.
  • Measurements ran on an optimized build of the test bundle; the commands are in Benchmarks/LiveTurns/README.md.

Not verified:

  • The setting in the running app with a microphone. Nothing was launched on this Mac. The replay above runs the preview's own code, but the test writes the two tracks.
  • The whole unit suite was not run locally, only the classes above. The hosted unit job runs it.
  • The Settings row was compiled, not looked at.
  • A real meeting, and any Mac but an M4 Max. One scripted call per language with synthetic voices; a slower Mac's delay grows by its decode time.
  • What showing the turn in progress costs in processor time and battery.
  • Windows and Linux: only the generated catalog changed; nothing was compiled or run there.
  • Translations of the two new strings (23 languages) are machine-made and unreviewed, like the catalogs' others.
  • RadarScreenAnswersTests/testAModelThatReadsPicturesIsSentTheCropAndItsCostCounts failed once in a combined run (it timed out after 10 s) and passed in three others, twice alone. It does not touch this code.

Changed test expectations

None. Every test in this PR is new.

Linked issues

None.

Risk / rollout notes

  • Off by default. With it off, the preview behaves as before; run() now shares its cursor setup and failure handling with the new mode, and the 24 existing LiveMeetingTranscriberTests pass unchanged.
  • Live Radar reads more often with it on: after each turn, at most every 3 s, where it read about every 20 s. With a remote Think model that is more requests, and with a local one more load during the meeting. This is why the setting is opt-in rather than the new default.
  • A transcription server as the Transcribe model gets one upload per turn where it got one per window, and never the turn in progress.
  • Parakeet reads Serbian wrongly today, with or without this setting: under the app's sr hint it writes Cyrillic and scored 81 to 84% word errors on the Serbian recording, against 26 to 30% with a Croatian hint. Not fixed here.
  • Mac only. The Windows and Linux preview is unchanged.
  • Scripts/ci/unit-skips.json lists the three opt-in measurements.

…peaker

The live transcript preview transcribes windows of 4 to 12 seconds, so a
turn's last words wait up to 4 s for their window to fill, and windows cut
mid-sentence. Meta's fast live text was the only way to get text right
after each turn.

A new setting, Turn-by-turn live text (Settings → Meetings → Processing,
off by default), does it with the Transcribe model on this Mac:

- LiveTurnCutter closes a turn after two quiet 256 ms chunks of speech
  detection, after one once the turn is 4 s long, or at 15 s at the
  quietest point of its last 1.5 s.
- LiveMeetingTranscriber.runTurns hears both live files every 100 ms, and
  transcribes closed turns one at a time. A failed turn is retried and
  survives a pause of the preview.
- While the model answers within 0.25 s and runs on this Mac, the turn
  still being spoken is transcribed again every 2 s and shown as a partial
  line. A transcription server hears each turn once, when it is finished.
- deliversTurns is true in this mode, so Live Radar reads after each turn.
  Muse's fast live text keeps precedence and is unchanged.

One read of the live file can stop short of the frames asked for; span
reads now loop until they have them.

The setting's two strings are in all 24 languages, and the Windows and
Linux catalogs are regenerated from them.
LiveTurnLatencyBenchmarkTests replays the bundled sample call, in English
and Serbian, through the app's engines as if it were being recorded:
turn by turn with LiveTurnCutter, in the preview's windows, through
FluidAudio's streaming recognizer, and in real time through
LiveMeetingTranscriber itself. Benchmarks/LiveTurns has the scripts, the
method and the results of 10 November 2026.

On an M4 Max with one track per side, a turn's text followed its last
word by a median 0.5 to 0.7 s (0.9 s at worst) with Parakeet, Granite
fast and Qwen3-ASR 1.7B, and 1.3 to 1.4 s with Whisper, against 1.7 to
3.2 s (3.8 to 5.0 s at worst) for windows, with fewer word errors and no
more CPU time.

The three tests are opt-in and listed in unit-skips.json.
@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
lokalbot 725e265 Commit Preview URL

Branch Preview URL
Nov 10 2026, 09:25 PM

@stevyhacker
stevyhacker marked this pull request as ready for review November 11, 2026 05:30
@stevyhacker
stevyhacker merged commit 08ebfe9 into master Nov 11, 2026
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant