Add Romanian TTS support via a new MMS engine - #916
Open
at-aepeak-com wants to merge 5 commits into
Open
Conversation
New 'mms' engine wrapping facebook/mms-tts-ron (Meta MMS, VITS architecture) — pure PyTorch via the already-pinned transformers dependency, zero new packages. Single preset voice, no cloning, modeled on the Kokoro preset-voice pattern. The mms-tts-ron vocab mixes Romanian diacritic conventions (comma-below s U+0219 but cedilla t U+0163), and the character-level VitsTokenizer silently drops out-of-vocab characters, so input text is NFC-normalized and both real-world variants are mapped onto the trained forms before tokenizing. Registered in the model config registry, engine factory, request-model regexes (language 'ro', engine 'mms'), preset-voice endpoints, and the PyInstaller hidden imports. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Romanian added to ALL_LANGUAGES, 'mms' wired through every engine enumeration: engine union types, zod schema, engine picker, display names, preset-engine sets (profile form/list/card, floating generate box), model management filter, and captures play-as union. MMS is treated as a preset-voice engine like Kokoro and Qwen CustomVoice. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Unit coverage for the Romanian diacritics normalization (cedilla vs comma-below variants, NFC composition, lossless handling of unknown chars), engine registration surfaces, and request-model regexes. End-to-end generation tests (model download ~150MB) are opt-in via VOICEBOX_MMS_E2E=1: Romanian text with both diacritic conventions produces valid 16kHz float32 audio, both spellings tokenize identically, seeded generation is deterministic, and fully out-of-vocab text falls back to silence. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
README engine/language counts (7 to 8 engines, 23 to 24 languages) and tables, preset-voices docs page with the MMS section, PROJECT_STATUS engine tracking, and a CHANGELOG entry under Unreleased. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (4)
🚧 Files skipped from review as they are similar to previous changes (3)
📝 WalkthroughWalkthroughAdds Romanian MMS-TTS as a new preset engine with backend inference, diacritic normalization, model and voice registration, frontend selection, request validation, tests, packaging, and documentation updates. ChangesMMS Romanian TTS
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant User
participant EngineModelSelector
participant useGenerationForm
participant BackendFactory
participant MMSTTSBackend
User->>EngineModelSelector: choose MMS Romanian
EngineModelSelector->>useGenerationForm: set engine to mms
useGenerationForm->>BackendFactory: submit mms-tts-ron request
BackendFactory->>MMSTTSBackend: create or reuse MMS backend
MMSTTSBackend-->>User: return Romanian waveform
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
app/src/components/CapturesTab/CapturesTab.tsx (1)
266-278: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winHandle MMS’s Romanian-only language constraint.
When a user plays a capture through an MMS profile, this code prefers
capture.language. A non-Romanian capture therefore sendsengine: 'mms'with (for example)en, even though MMS only supports Romanian. Reject this combination in the UI or explicitly validate against the selected profile language before callinggenerateSpeech.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@app/src/components/CapturesTab/CapturesTab.tsx` around lines 266 - 278, Update the capture playback flow around the engine selection and generateSpeech call to prevent MMS from being used with any non-Romanian language. Validate the resolved language against the selected voice profile before calling generateSpeech, and reject the request through the existing UI error path rather than sending an invalid engine/language combination.
🧹 Nitpick comments (1)
backend/backends/mms_backend.py (1)
14-15: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDocstring overstates how easy adding a language is.
The module doc says adding a language is "a one-line addition to
MMS_HF_REPOS," but_get_model_pathand_load_model_syncalways resolve toMMS_HF_REPOS[MMS_DEFAULT_LANGUAGE]rather than any language actually requested — a newMMS_HF_REPOSentry wouldn't be reachable without further code changes to thread the language through model loading.✏️ Clarify the docstring (or parameterize by language)
-Adding a language is a one-line addition to ``MMS_HF_REPOS`` plus a voice -entry in ``MMS_VOICES`` and a ``ModelConfig`` registration. +Adding a language currently requires updating ``MMS_HF_REPOS``, ``MMS_VOICES``, +a ``ModelConfig`` registration, *and* threading the target language through +``_get_model_path``/``_load_model_sync`` (which are hardcoded to +``MMS_DEFAULT_LANGUAGE`` today).Also applies to: 98-99, 111-124
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@backend/backends/mms_backend.py` around lines 14 - 15, Correct the module docstring to stop claiming that adding a language only requires one MMS_HF_REPOS entry, and describe the additional model-loading changes currently required. Update the documentation near _get_model_path and _load_model_sync consistently, unless you instead parameterize those functions to resolve MMS_HF_REPOS using the requested language and preserve the existing default-language behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@backend/models.py`:
- Line 21: Extract the duplicated language-code and engine regex patterns into
shared constants in the models module, then replace the inline patterns in
VoiceProfileCreate, GenerationRequest, MCPClientBindingResponse,
MCPClientBindingUpsert, and SpeakRequest with those constants. Preserve the
current accepted values, including ro and mms, so all validation surfaces remain
consistent.
In `@README.md`:
- Around line 75-77: Update the README preset-voices bullet to mention the
Romanian MMS preset voice alongside Kokoro and Qwen CustomVoice, while
preserving the existing voice-cloning and preset-voice descriptions.
---
Outside diff comments:
In `@app/src/components/CapturesTab/CapturesTab.tsx`:
- Around line 266-278: Update the capture playback flow around the engine
selection and generateSpeech call to prevent MMS from being used with any
non-Romanian language. Validate the resolved language against the selected voice
profile before calling generateSpeech, and reject the request through the
existing UI error path rather than sending an invalid engine/language
combination.
---
Nitpick comments:
In `@backend/backends/mms_backend.py`:
- Around line 14-15: Correct the module docstring to stop claiming that adding a
language only requires one MMS_HF_REPOS entry, and describe the additional
model-loading changes currently required. Update the documentation near
_get_model_path and _load_model_sync consistently, unless you instead
parameterize those functions to resolve MMS_HF_REPOS using the requested
language and preserve the existing default-language behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 5204a75c-b666-41bc-b29d-661591e604f4
📒 Files selected for processing (22)
CHANGELOG.mdREADME.mdapp/src/components/CapturesTab/CapturesTab.tsxapp/src/components/Generation/EngineModelSelector.tsxapp/src/components/Generation/FloatingGenerateBox.tsxapp/src/components/ServerSettings/ModelManagement.tsxapp/src/components/VoiceProfiles/ProfileCard.tsxapp/src/components/VoiceProfiles/ProfileForm.tsxapp/src/components/VoiceProfiles/ProfileList.tsxapp/src/lib/api/types.tsapp/src/lib/constants/languages.tsapp/src/lib/hooks/useGenerationForm.tsapp/src/lib/utils/format.tsbackend/backends/__init__.pybackend/backends/mms_backend.pybackend/build_binary.pybackend/models.pybackend/routes/profiles.pybackend/services/profiles.pybackend/tests/test_mms_backend.pydocs/PROJECT_STATUS.mddocs/content/docs/overview/preset-voices.mdx
- CapturesTab: only honor a capture's language when the target engine supports it (MMS is Romanian-only; also hardens Kokoro playback), falling back to the profile's language otherwise - models.py: extract shared TTS_LANGUAGE_PATTERN / TTS_ENGINE_PATTERN constants to replace seven duplicated inline regexes - mms_backend: correct module docstring — adding a language also needs the checkpoint resolution threaded through model loading - README: mention MMS (Romanian) in the preset-voices bullet Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #578
What
New
mmsTTS engine wrappingfacebook/mms-tts-ron(Meta MMS, VITS architecture), adding Romanian as Voicebox's 24th language. Single preset voice, no cloning — follows the existing Kokoro preset-engine pattern (kokoro_backend.pywas the structural template).Zero new dependencies — uses the already-pinned
transformers. ~145 MB model, CPU-realtime, 16 kHz output, works on all platforms (no CUDA/platform gating needed).Notable: Romanian diacritics handling
The MMS Romanian tokenizer's vocab is mixed: it contains comma-below ș (U+0219) but cedilla ţ (U+0163), and silently drops the other two variants (ş U+015F, ț U+021B) — the tokenizer strips characters not in its vocab with no error. Real-world Romanian text freely mixes both spellings, so without handling this, letters silently vanish from the audio.
The backend NFC-normalizes input and maps each diacritic to the form the checkpoint was trained on (ş→ș, ț→ţ, uppercase included). An e2e test proves both spellings produce identical token ids with zero dropped characters.
Extensible by design
MMS_HF_REPOSis a module-level{language: checkpoint}map — further MMS languages are one-line additions (e.g. Croatianmms-tts-hrv, requested in #549). Only Romanian ships in this PR to keep it reviewable.Changes
backend/backends/mms_backend.py(new) —MMSTTSBackendimplementing theTTSBackendprotocolTTS_ENGINES,ModelConfig, factory branch inbackends/__init__.py;ro/mmsadded to validation regexes inmodels.py; preset-voice branches inroutes/profiles.py+services/profiles.py; PyInstaller hidden-import inbuild_binary.pyroinALL_LANGUAGES,mmsinENGINE_LANGUAGES, engine wired through the selector, profile forms, model management, and format/display mapspreset-voices.mdx,PROJECT_STATUS.md, CHANGELOGTesting
VOICEBOX_MMS_E2E=1(real model download + generation: 16 kHz float32, no NaNs; token-identity proof for both diacritic spellings; seeded determinism)bunx biome check/ruff checkclean on touched files (no new findings)🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Documentation
Tests