Skip to content

Time words with the CoreML Qwen3 forced aligner - #504

Merged
stevyhacker merged 3 commits into
masterfrom
claude/project-thread-8lkkf4
Nov 16, 2026
Merged

stevyhacker merged 3 commits into
masterfrom
claude/project-thread-8lkkf4

Conversation

@stevyhacker

Copy link
Copy Markdown
Owner

Requested by Stevan · project thread

Summary

Word timing for speaker attribution moves from the MLX 4-bit Qwen3 forced aligner to speech-swift's CoreML export of the same model (aufklarer/Qwen3-ForcedAligner-0.6B-CoreML-INT8, pinned file by file). On the same text it attributes speakers as well as MLX, and its peak memory is about a quarter of MLX's. It also clears the last thing blocking a move off MLX for word timing. The CoreML encoder takes at most 30 s of audio, so the Qwen3-ASR 1.7B word-attribution windows are capped at 30 s instead of 60 s. On AMI the two caps are within noise.

Benchmark: Benchmarks/transcribe/QwenSpanLength/results/2026-11-15/REPORT.md

Condition AMI WER AMI cpWER cpWER vs MLX, same text (95% CI) Speaker memory (AMI, meetings) Failed windows Speed Peak memory
MLX 4-bit, ≤60 s (shipped) 17.81% 34.23% — 360/417, 1406/1501 0/187 223× 4.07 GB
MLX 4-bit, ≤30 s 17.95% 33.85% — 361/417, 1422/1501 0/302 210× 3.51 GB
CoreML INT8, ≤30 s (this PR) 17.95% 33.75% −0.10 (−0.58…+0.36) 368/417, 1419/1501 0/302 126× 0.89 GB
CoreML FP16, ≤30 s 17.95% 33.68% −0.17 (−0.57…+0.22) 366/417, 1413/1501 0/302 107× 0.83 GB

Validation

  • Full unit suite on an M-series Mac: 4,983 tests, 0 failures, 56 opt-in skips. Two tests that are known to hang locally were excluded, as in earlier local runs: LlamaCotypingRuntimeTests and one CotypingMemoryRelevanceTests test.
  • The two opt-in aligner tests pass against the real INT8 download. The pinned files verify offline, and the footprint stays flat over 40 passes. The footprint test fails without the per-pass autorelease pool.
  • swiftlint clean.
  • Not run: an end-to-end --process run with a dev build, UI tests (CI), and meeting accuracy against the vendor consensus. The consensus transcripts were deleted, and rebuilding them would mean uploading meeting audio again.

Changed test expectations

  • WordAttributionTests window expectation for 1.7B: maxSeconds 60 → 30. The old value was right for the MLX aligner. The CoreML encoder cannot take more than 30 s (3,000 mel frames), and the benchmark shows ≤30 s within noise of ≤60 s on AMI (WER +0.14, −0.55…+0.72; cpWER −0.38, −1.31…+0.44).
  • WordAttributionTests snapshot expectations: repository, file set and size (1.0 GB → 1.08 GB) now describe the CoreML INT8 snapshot instead of the MLX 4-bit one.
  • testCachedAlignerKeepsScratchBuffersBoundedAcrossSegmentLengths → testCachedAlignerKeepsFootprintFlatAcrossSegments. It checked MLX's buffer pool, which the aligner no longer uses, and now checks the process footprint, where the CoreML leak showed.

Linked issues

None.

Risk / rollout notes

  • Transcripts change: 1.7B decode windows are now ≤30 s. The 30 Sep report measured ≤30 s about 0.26 points behind ≤60 s on meeting agreement, and that wasn't re-measured here. The transcript cache key moves to merged-windows-v3.
  • New download: about 1.08 GB for the CoreML model. The old MLX aligner folder (~1 GB) is deleted, by exact path, once the new snapshot is verified. Unit tests never touch it.
  • First load: the first load on a Mac takes about 13 s while the Neural Engine specializes the model. macOS caches that, and later loads take 0.6 s.
  • Thin token headroom: the densest benchmark window used 743 of the 768 tokens the decoder takes. A segment that overflows falls back to one speaker at its midpoint, and the fallback is logged.
  • Serbian unchanged: Serbian still isn't in the aligner's language list, so it keeps speaker-region attribution.
  • MLX still linked: the Qwen3ASR module still links MLX. Removing MLX entirely needs that target split in the speech-swift fork, and the Qwen3-ASR and Granite Turbo engines moved off it.

Generated by Claude Code

stevyhacker and others added 2 commits November 15, 2026 07:07
The span-length harness now builds against the app's current pins (the
speech-swift fork at 8e87821, FluidAudio 0.18.3, mlx-swift 0.32.3), and its
align mode can run CoreMLForcedAligner, recording each window's time and
failure reason plus load time and peak footprint. score_coreml.py compares
the shipped MLX layout, MLX on 30 s windows and the INT8 and FP16 CoreML
exports, and diffs word boundaries on identical text.

On identical 30 s text, CoreML INT8 is within noise of MLX on AMI cpWER
(-0.10, -0.58..+0.36) and speaker-memory turns, never failed, and peaks at
0.9 GB against 3.5-4.1 GB, at 126x real time against 210x. It needs an
autorelease pool per pass, or the footprint grows to 4.6 GB over a run.

CloudSTT's score.py gets the spelling-list path fix for the repository
restructure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QD3UkweUq739ov9W6LeL2F
QwenWordAligner now runs speech-swift's CoreMLForcedAligner on
aufklarer/Qwen3-ForcedAligner-0.6B-CoreML-INT8 (pinned per file, compiled
.mlmodelc bundles only) instead of the MLX 4-bit aligner. On identical text
it attributes as well (AMI cpWER -0.10, -0.58..+0.36) at about a quarter of
the memory (0.9 GB against 3.5-4.1 GB peak footprint). Each pass runs in an
autorelease pool: without one, CoreML's outputs grew a long track's
footprint from 0.8 GB to 4.6 GB.

The encoder takes at most 30 s, so Qwen3-ASR 1.7B's merged word-attribution
windows drop from 60 s to 30 s (within noise on AMI). Longer segments are
refused before the model loads, and any rejected segment keeps one speaker
at its midpoint, as before. The old MLX aligner folder is removed once the
new snapshot is verified (never from the unit-test host), MLX scratch
trimming goes, the RAM estimate is 0.9 GiB, and the aligned-words checkpoint
key moves to v3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QD3UkweUq739ov9W6LeL2F
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
lokalbot d2aeacd Commit Preview URL

Branch Preview URL
Nov 16 2026, 11:47 AM

@stevyhacker stevyhacker self-assigned this Nov 16, 2026

Copy link
Copy Markdown
Owner Author

UI build and critical tests failed on RedesignUITests.testActionSearchPreservesSelectionAndCompletionRecoversFromSaveFailure at RedesignUITests.swift:837. That line waits 4 s for the "Could not update this action" error after a blocked save.

I don't think this PR causes it:

  • The diff doesn't touch the Actions screen, the outcome state store or the UI test host.
  • The same test failed at the same line on Release 0.13.1 #502 (claude/release-0.13.1, run 43634023243) on 2026-11-14.

I'll re-run the failed job once the rest of the workflow finishes.


Generated by Claude Code

Keeps master's llama.cpp Qwen3-ASR 1.7B (legacy MLX repository constant,
its own port, unit skip) with this branch's CoreML forced aligner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QD3UkweUq739ov9W6LeL2F
@stevyhacker
stevyhacker merged commit 60c4954 into master Nov 16, 2026
8 checks passed
@stevyhacker stevyhacker mentioned this pull request Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants