Repository navigation
Time words with the CoreML Qwen3 forced aligner - #504
Merged
Merged
Conversation
The span-length harness now builds against the app's current pins (the speech-swift fork at 8e87821, FluidAudio 0.18.3, mlx-swift 0.32.3), and its align mode can run CoreMLForcedAligner, recording each window's time and failure reason plus load time and peak footprint. score_coreml.py compares the shipped MLX layout, MLX on 30 s windows and the INT8 and FP16 CoreML exports, and diffs word boundaries on identical text. On identical 30 s text, CoreML INT8 is within noise of MLX on AMI cpWER (-0.10, -0.58..+0.36) and speaker-memory turns, never failed, and peaks at 0.9 GB against 3.5-4.1 GB, at 126x real time against 210x. It needs an autorelease pool per pass, or the footprint grows to 4.6 GB over a run. CloudSTT's score.py gets the spelling-list path fix for the repository restructure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QD3UkweUq739ov9W6LeL2F
QwenWordAligner now runs speech-swift's CoreMLForcedAligner on aufklarer/Qwen3-ForcedAligner-0.6B-CoreML-INT8 (pinned per file, compiled .mlmodelc bundles only) instead of the MLX 4-bit aligner. On identical text it attributes as well (AMI cpWER -0.10, -0.58..+0.36) at about a quarter of the memory (0.9 GB against 3.5-4.1 GB peak footprint). Each pass runs in an autorelease pool: without one, CoreML's outputs grew a long track's footprint from 0.8 GB to 4.6 GB. The encoder takes at most 30 s, so Qwen3-ASR 1.7B's merged word-attribution windows drop from 60 s to 30 s (within noise on AMI). Longer segments are refused before the model loads, and any rejected segment keeps one speaker at its midpoint, as before. The old MLX aligner folder is removed once the new snapshot is verified (never from the unit-test host), MLX scratch trimming goes, the RAM estimate is 0.9 GiB, and the aligned-words checkpoint key moves to v3. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QD3UkweUq739ov9W6LeL2F
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
lokalbot | d2aeacd | Commit Preview URL Branch Preview URL |
Nov 16 2026, 11:47 AM |
Owner
Author
|
UI build and critical tests failed on I don't think this PR causes it:
I'll re-run the failed job once the rest of the workflow finishes. Generated by Claude Code |
Keeps master's llama.cpp Qwen3-ASR 1.7B (legacy MLX repository constant, its own port, unit skip) with this branch's CoreML forced aligner. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QD3UkweUq739ov9W6LeL2F
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Stevan · project thread
Summary
Word timing for speaker attribution moves from the MLX 4-bit Qwen3 forced aligner to speech-swift's CoreML export of the same model (
aufklarer/Qwen3-ForcedAligner-0.6B-CoreML-INT8, pinned file by file). On the same text it attributes speakers as well as MLX, and its peak memory is about a quarter of MLX's. It also clears the last thing blocking a move off MLX for word timing. The CoreML encoder takes at most 30 s of audio, so the Qwen3-ASR 1.7B word-attribution windows are capped at 30 s instead of 60 s. On AMI the two caps are within noise.Benchmark:
Benchmarks/transcribe/QwenSpanLength/results/2026-11-15/REPORT.mdValidation
LlamaCotypingRuntimeTestsand oneCotypingMemoryRelevanceTeststest.swiftlintclean.--processrun with a dev build, UI tests (CI), and meeting accuracy against the vendor consensus. The consensus transcripts were deleted, and rebuilding them would mean uploading meeting audio again.Changed test expectations
WordAttributionTestswindow expectation for 1.7B:maxSeconds 60→30. The old value was right for the MLX aligner. The CoreML encoder cannot take more than 30 s (3,000 mel frames), and the benchmark shows ≤30 s within noise of ≤60 s on AMI (WER +0.14, −0.55…+0.72; cpWER −0.38, −1.31…+0.44).WordAttributionTestssnapshot expectations: repository, file set and size (1.0 GB → 1.08 GB) now describe the CoreML INT8 snapshot instead of the MLX 4-bit one.testCachedAlignerKeepsScratchBuffersBoundedAcrossSegmentLengths→testCachedAlignerKeepsFootprintFlatAcrossSegments. It checked MLX's buffer pool, which the aligner no longer uses, and now checks the process footprint, where the CoreML leak showed.Linked issues
None.
Risk / rollout notes
merged-windows-v3.Qwen3ASRmodule still links MLX. Removing MLX entirely needs that target split in the speech-swift fork, and the Qwen3-ASR and Granite Turbo engines moved off it.Generated by Claude Code