feat(speech): add pocket-tts CPU text-to-speech pipeline - #2350
Open
sandyz1000 wants to merge 9 commits into
Open
sandyz1000 wants to merge 9 commits into
sandyz1000 wants to merge 9 commits into
Conversation
Code Metrics Report━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Language Files Lines Code Comments Blanks ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ C Header 46 9600 7138 1327 1135 C++ 1 36 30 1 5 C++ Header 6 4022 3237 399 386 CSS 5 1069 918 6 145 CUDA 143 38519 32957 1775 3787 Dockerfile 1 35 19 9 7 HTML 3 125 119 0 6 JavaScript 8 1241 1186 11 44 Jinja2 7 694 656 5 33 JSON 34 26374 26371 0 3 Makefile 1 18 16 0 2 MDX 37 6975 0 5157 1818 Metal Shading Lan| 37 14422 11414 1136 1872 PowerShell 1 657 571 31 55 Python 161 34938 31448 487 3003 Shell 3 1071 852 115 104 Plain Text 54 10714 0 9231 1483 TOML 28 1399 1212 43 144 TypeScript 11 1658 1418 66 174 YAML 3 25 23 2 0 ───────────────────────────────────────────────────────────────────────────────── Jupyter Notebooks 3 122 83 23 16 |- Markdown 1 60 30 22 8 |- Python 1 122 113 1 8 (Total) 304 226 46 32 ───────────────────────────────────────────────────────────────────────────────── Markdown 284 13126 0 9834 3292 |- BASH 27 349 268 48 33 |- Dockerfile 2 14 12 0 2 |- JSON 6 292 292 0 0 |- PowerShell 1 1 1 0 0 |- Python 135 7349 6119 306 924 |- Rust 63 3873 2873 398 602 |- TOML 7 116 97 0 19 (Total) 25120 9662 10586 4872 ───────────────────────────────────────────────────────────────────────────────── Rust 774 501441 457490 5598 38353 |- Markdown 445 10408 452 8767 1189 (Total) 511849 457942 14365 39542 ───────────────────────────────────────────────────────────────────────────────── Svelte 19 1974 1832 50 92 |- CSS 1 4 4 0 0 |- JavaScript 19 928 773 25 130 (Total) 2906 2609 75 222 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Total 1670 693771 590024 44873 58874 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds pocket-tts as a second speech pipeline alongside Dia, over the same OpenAI
/v1/audio/speechcontract. It's a Kyutai-style Mimi codec + FlowLM transformer that runs fast on CPU, so one mistral.rs binary can serve an LLM plus a CPU-fast TTS (Dia is GPU-oriented).Selectable via serve -m (auto-detected), serve speech, or from-config; voice defaults to alba and is settable via --voice / config / SDK.
/v1/audio/speechnow routes by the request's model, so pocket-tts (CPU) and Dia (GPU) can be loaded together and picked per request.Attribution
Model + inference code under
mistralrs-core/src/speech_models/pockettts/is ported from the pocket-tts Rust crate: https://github.com/babybirdprd/pocket-tts — mostly near-verbatim, recompiled against this repo's candle 0.11 (upstream pins 0.9.1). The rest is the mistral.rs integration layer.Source is MIT OR Apache-2.0, this repo is MIT — compatible. Attribution is also in the module header. Opening transparently so maintainers (and the pocket-tts author, if they wish) can review provenance.
Verification
cargo fmt --all --checkpass.