Skip to content

feat(speech): add pocket-tts CPU text-to-speech pipeline - #2350

Open
sandyz1000 wants to merge 9 commits into
EricLBuehler:masterfrom
sandyz1000:master
Open

sandyz1000 wants to merge 9 commits into
EricLBuehler:masterfrom
sandyz1000:master

Conversation

@sandyz1000

Copy link
Copy Markdown

Summary

Adds pocket-tts as a second speech pipeline alongside Dia, over the same OpenAI /v1/audio/speech contract. It's a Kyutai-style Mimi codec + FlowLM transformer that runs fast on CPU, so one mistral.rs binary can serve an LLM plus a CPU-fast TTS (Dia is GPU-oriented).

Selectable via serve -m (auto-detected), serve speech, or from-config; voice defaults to alba and is settable via --voice / config / SDK. /v1/audio/speech now routes by the request's model, so pocket-tts (CPU) and Dia (GPU) can be loaded together and picked per request.

Attribution

Model + inference code under mistralrs-core/src/speech_models/pockettts/ is ported from the pocket-tts Rust crate: https://github.com/babybirdprd/pocket-tts — mostly near-verbatim, recompiled against this repo's candle 0.11 (upstream pins 0.9.1). The rest is the mistral.rs integration layer.

Source is MIT OR Apache-2.0, this repo is MIT — compatible. Attribution is also in the module header. Opening transparently so maintainers (and the pocket-tts author, if they wish) can review provenance.

Verification

  • cargo check clean (0 warnings) and cargo fmt --all --check pass.
  • End-to-end on Apple Metal and over HTTP: kyutai/pocket-tts-without-voice-cloning produces healthy 24 kHz mono speech.

@sandyz1000 sandyz1000 changed the title Title: feat(speech): add pocket-tts CPU text-to-speech pipeline feat(speech): add pocket-tts CPU text-to-speech pipeline Jul 21, 2026
@github-actions

github-actions Bot commented Jul 21, 2026 •

Copy link
Copy Markdown
Code Metrics Report
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Language              Files        Lines         Code     Comments       Blanks
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 C Header                 46         9600         7138         1327         1135
 C++                       1           36           30            1            5
 C++ Header                6         4022         3237          399          386
 CSS                       5         1069          918            6          145
 CUDA                    143        38519        32957         1775         3787
 Dockerfile                1           35           19            9            7
 HTML                      3          125          119            0            6
 JavaScript                8         1241         1186           11           44
 Jinja2                    7          694          656            5           33
 JSON                     34        26374        26371            0            3
 Makefile                  1           18           16            0            2
 MDX                      37         6975            0         5157         1818
 Metal Shading Lan|       37        14422        11414         1136         1872
 PowerShell                1          657          571           31           55
 Python                  161        34938        31448          487         3003
 Shell                     3         1071          852          115          104
 Plain Text               54        10714            0         9231         1483
 TOML                     28         1399         1212           43          144
 TypeScript               11         1658         1418           66          174
 YAML                      3           25           23            2            0
─────────────────────────────────────────────────────────────────────────────────
 Jupyter Notebooks         3          122           83           23           16
 |- Markdown               1           60           30           22            8
 |- Python                 1          122          113            1            8
 (Total)                              304          226           46           32
─────────────────────────────────────────────────────────────────────────────────
 Markdown                284        13126            0         9834         3292
 |- BASH                  27          349          268           48           33
 |- Dockerfile             2           14           12            0            2
 |- JSON                   6          292          292            0            0
 |- PowerShell             1            1            1            0            0
 |- Python               135         7349         6119          306          924
 |- Rust                  63         3873         2873          398          602
 |- TOML                   7          116           97            0           19
 (Total)                            25120         9662        10586         4872
─────────────────────────────────────────────────────────────────────────────────
 Rust                    774       501441       457490         5598        38353
 |- Markdown             445        10408          452         8767         1189
 (Total)                           511849       457942        14365        39542
─────────────────────────────────────────────────────────────────────────────────
 Svelte                   19         1974         1832           50           92
 |- CSS                    1            4            4            0            0
 |- JavaScript            19          928          773           25          130
 (Total)                             2906         2609           75          222
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Total                  1670       693771       590024        44873        58874
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant