feat: add OpenMOSS XY-Tokenizer (16 kHz → 24 kHz neural codec)

XY-Tokenizer is a Whisper-style audio codec from OpenMOSS-Team / MOSS-TTSD.
End-to-end pipeline:

  16 kHz mono PCM
   → Whisper-style mel-fbank (n_fft=400, hop=160, 80 mels, Slaney scale)
   → parallel semantic + acoustic OmniAudioEncoder (2× conv stride-2 + 12-layer
     Whisper transformer with q/v/out biases, k bias-free, sinusoidal PE)
   → semantic_encoder_adapter (4-layer Whisper transformer)
   → concat[sem_adapter, acoust] (1536 ch)
   → pre_rvq_adapter (Linear 1536→768 + 4-layer transformer)
   → ResidualDownConv (conv-gated SiLU pool, avg=4) → 3072 ch
   → quantizer.input_proj (1×1 WNConv, weight-norm baked at convert)
   → 8-level Euclidean-NN Residual VQ (1024×512 codebook with pre-baked
     `codebook_sq_norm` so `argmin ||z - cb[i]||^2` becomes one matmul + bias)
   → quantizer.output_proj
   → post_rvq_adapter (Linear 3072→768 + 4 layers + Linear 768→3072)
   → UpConv (ConvTranspose1d stride=4)
   → 12-layer Whisper-style mel decoder + deconv1 (s=2) + deconv2 (s=1)
   → enhanced Vocos backbone (embed conv 80→512 k=7, initial LN,
     30 ConvNeXt blocks, final LN, head Linear 512→962)
   → CPU iSTFT (n_fft=960, hop=240, periodic Hann) → 24 kHz mono audio

Encoder threads `n_valid` through every transformer block to mirror HF's
`attention_mask`-based SDPA bias, and uses GELU-erf (matching `F.gelu`)
throughout — without those two pieces the encoder body diverges from HF
and codes drift.  `tests/e2e/xy_tokenizer_decode_smoke.py` verifies all
three parts: cpp decode of HF codes ≥ 0.95 corr, cpp encode codes ≥ 95 %
match HF, cpp e2e ≥ 0.95 corr vs HF reconstruction.  On the smoke clip:

  - cpp decode(HF codes) vs HF rec: corr 0.97
  - cpp encode codes vs HF codes: 100 % match
  - cpp e2e vs HF e2e: corr 0.97

On a longer (3 s) clip the encoder hits 295/296 = 99.66 % code match and
the e2e hits corr 0.975 against the HF reference.

Mel-fbank is computed CPU-side (Slaney triangular filterbank, log10 +
global max-clamp + (x+4)/4 normalize, reflection-padded center-True STFT
to match HF's WhisperFeatureExtractor).  Everything else runs as a single
ggml graph per public call.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: hans00

CMakeLists.txt

@@ -63,6 +63,7 @@ add_library(codec STATIC
     src/models/xcodec2.cpp
     src/models/snac.cpp
     src/models/moss_audio.cpp
+    src/models/xy_tokenizer.cpp
 )
 
 target_include_directories(codec

include/codec.h

@@ -26,6 +26,7 @@ enum codec_arch {
     CODEC_ARCH_XCODEC2 = 11,
     CODEC_ARCH_SNAC = 12,
     CODEC_ARCH_MOSS_AUDIO = 13,
+    CODEC_ARCH_XY_TOKENIZER = 14,
 };
 
 enum codec_status {

scripts/convert-to-gguf.py

@@ -64,6 +64,8 @@ def detect_model_type_from_config(config_path: Path) -> str:
         return "snac"
     elif "moss-audio" in model_type or "moss_audio" in model_type or "mossaudio" in model_type:
         return "moss_audio"
+    elif "xy_tokenizer" in model_type or "xy-tokenizer" in model_type:
+        return "xy_tokenizer"
     else:
         # Try to infer from architecture or other fields
         arch = config.get("architectures", [""])[0].lower() if config.get("architectures") else ""
@@ -93,6 +95,8 @@ def detect_model_type_from_config(config_path: Path) -> str:
             return "snac"
         elif "moss" in arch and "audio" in arch:
             return "moss_audio"
+        elif "xy_tokenizer" in arch or "xytokenizer" in arch:
+            return "xy_tokenizer"
 
     raise ValueError(f"Unknown model type: {model_type}. Cannot auto-detect.")
 
@@ -126,6 +130,8 @@ def infer_model_type_from_filename(filename: str) -> str | None:
         return 'snac'
     elif 'moss' in name_lower and 'audio' in name_lower:
         return 'moss_audio'
+    elif 'xy_tokenizer' in name_lower or 'xy-tokenizer' in name_lower:
+        return 'xy_tokenizer'
     return None
 
 

scripts/converters/__init__.py

@@ -14,6 +14,7 @@
 from .xcodec2 import XCodec2Converter
 from .snac import SnacConverter
 from .moss_audio import MossAudioConverter
+from .xy_tokenizer import XYTokenizerConverter
 
 # Registry of supported models
 _CONVERTER_REGISTRY = {
@@ -31,6 +32,7 @@
     'snac': SnacConverter,
     'moss_audio': MossAudioConverter,
     'moss_audio_nano': MossAudioConverter,
+    'xy_tokenizer': XYTokenizerConverter,
 }
 
 

scripts/converters/xy_tokenizer.py

@@ -13,6 +13,7 @@
 #include "models/xcodec2.h"
 #include "models/snac.h"
 #include "models/moss_audio.h"
+#include "models/xy_tokenizer.h"
 #include "ops/safe_math.h"
 #include "runtime/graph.h"
 #include "runtime/gguf_kv.h"
@@ -133,6 +134,9 @@ enum codec_arch codec_arch_from_string(const std::string & arch) {
     if (arch == "moss_audio_tokenizer" || arch == "moss-audio-tokenizer" || arch == "moss_audio") {
         return CODEC_ARCH_MOSS_AUDIO;
     }
+    if (arch == "xy_tokenizer" || arch == "xy-tokenizer") {
+        return CODEC_ARCH_XY_TOKENIZER;
+    }
 
     return CODEC_ARCH_UNKNOWN;
 }
@@ -165,6 +169,8 @@ static const codec_model_vtable * codec_model_vtable_for_arch(enum codec_arch ar
             return codec_snac_vtable();
         case CODEC_ARCH_MOSS_AUDIO:
             return codec_moss_audio_vtable();
+        case CODEC_ARCH_XY_TOKENIZER:
+            return codec_xy_tokenizer_vtable();
         case CODEC_ARCH_UNKNOWN:
         default:
             return nullptr;
@@ -186,6 +192,7 @@ const char * codec_arch_name(enum codec_arch arch) {
         case CODEC_ARCH_XCODEC2:            return "XCodec2";
         case CODEC_ARCH_SNAC:               return "SNAC";
         case CODEC_ARCH_MOSS_AUDIO:         return "MOSS-Audio-Tokenizer";
+        case CODEC_ARCH_XY_TOKENIZER:       return "XY-Tokenizer";
         case CODEC_ARCH_UNKNOWN:
         default:                            return "unknown";
     }