data_transforms dataloader block for automatically RVCing inputs

.gitignore

@@ -94,3 +94,4 @@ package-lock.json
 site/
 tunableop_results*.csv
 .cog/
+artifacts/

cog.yaml

@@ -9,10 +9,15 @@ build:
     - libsndfile1
   run:
     - rm -rf SimpleTuner && git clone --depth 1 --branch main https://github.com/bghira/SimpleTuner.git SimpleTuner
+    - python -c "from pathlib import Path; p=Path('SimpleTuner/simpletuner/helpers/multiaspect/dataset.py'); s=p.read_text(); old='if model_family != \"ace_step\" and \"target_size\" in image_metadata:'; new='if (\\n                model_family != \"ace_step\"\\n                and image_metadata.get(\"dataset_type\") != \"audio\"\\n                and \"target_size\" in image_metadata\\n            ):'; assert old in s or new in s; p.write_text(s.replace(old, new))"
+    - python -c 'from pathlib import Path; import base64; p=Path("SimpleTuner/simpletuner/helpers/models/minimaxmusic/model.py"); s=p.read_text(); old=base64.b64decode("ICAgICAgICAgICAgY2FwdGlvbiA9IGV4YW1wbGUuZ2V0KCJwcm9tcHQiKSBvciBleGFtcGxlLmdldCgidGFncyIpCiAgICAgICAgICAgIGx5cmljcyA9IGV4YW1wbGUuZ2V0KCJseXJpY3MiKQogICAgICAgICAgICBpZiBub3QgaXNpbnN0YW5jZShjYXB0aW9uLCBzdHIpIG9yIG5vdCBjYXB0aW9uLnN0cmlwKCk6CiAgICAgICAgICAgICAgICByYWlzZSBWYWx1ZUVycm9yKCJNaW5pTWF4IE11c2ljIDMgbGFuZ3VhZ2UgbW9kZWwgdHJhaW5pbmcgcmVxdWlyZXMgJ3Byb21wdCcgKG9yICd0YWdzJykgbWV0YWRhdGEuIikKICAgICAgICAgICAgaWYgbm90IGlzaW5zdGFuY2UobHlyaWNzLCBzdHIpOgogICAgICAgICAgICAgICAgcmFpc2UgVmFsdWVFcnJvcigKICAgICAgICAgICAgICAgICAgICAiTWluaU1heCBNdXNpYyAzIGxhbmd1YWdlIG1vZGVsIHRyYWluaW5nIHJlcXVpcmVzICdseXJpY3MnIG1ldGFkYXRhIChhbiBlbXB0eSBzdHJpbmcgaXMgIgogICAgICAgICAgICAgICAgICAgICJhbGxvd2VkIGZvciBpbnN0cnVtZW50YWwgb3IgcmVndWxhcmlzYXRpb24gdHJhY2tzKS4iCiAgICAgICAgICAgICAgICApCg==").decode(); new=base64.b64decode("ICAgICAgICAgICAgY2FwdGlvbiA9IE5vbmUKICAgICAgICAgICAgY2FwdGlvbl9wcmVzZW50ID0gRmFsc2UKICAgICAgICAgICAgZm9yIGtleSBpbiAoInByb21wdCIsICJ0YWdzIiwgImluc3RhbmNlX3Byb21wdF90ZXh0Iik6CiAgICAgICAgICAgICAgICBpZiBrZXkgbm90IGluIGV4YW1wbGU6CiAgICAgICAgICAgICAgICAgICAgY29udGludWUKICAgICAgICAgICAgICAgIGNhbmRpZGF0ZSA9IGV4YW1wbGVba2V5XQogICAgICAgICAgICAgICAgaWYgY2FuZGlkYXRlIGlzIE5vbmU6CiAgICAgICAgICAgICAgICAgICAgY29udGludWUKICAgICAgICAgICAgICAgIGNhcHRpb24gPSBjYW5kaWRhdGUKICAgICAgICAgICAgICAgIGNhcHRpb25fcHJlc2VudCA9IFRydWUKICAgICAgICAgICAgICAgIGJyZWFrCiAgICAgICAgICAgIGx5cmljcyA9IGV4YW1wbGUuZ2V0KCJseXJpY3MiKQogICAgICAgICAgICBpZiBub3QgY2FwdGlvbl9wcmVzZW50IG9yIG5vdCBpc2luc3RhbmNlKGNhcHRpb24sIHN0cik6CiAgICAgICAgICAgICAgICByYWlzZSBWYWx1ZUVycm9yKCJNaW5pTWF4IE11c2ljIDMgbGFuZ3VhZ2UgbW9kZWwgdHJhaW5pbmcgcmVxdWlyZXMgJ3Byb21wdCcgKG9yICd0YWdzJykgbWV0YWRhdGEuIikKICAgICAgICAgICAgaWYgbHlyaWNzIGlzIE5vbmU6CiAgICAgICAgICAgICAgICBseXJpY3MgPSAiIgogICAgICAgICAgICBlbGlmIG5vdCBpc2luc3RhbmNlKGx5cmljcywgc3RyKToKICAgICAgICAgICAgICAgIHJhaXNlIFZhbHVlRXJyb3IoCiAgICAgICAgICAgICAgICAgICAgIk1pbmlNYXggTXVzaWMgMyBsYW5ndWFnZSBtb2RlbCB0cmFpbmluZyByZXF1aXJlcyAnbHlyaWNzJyBtZXRhZGF0YSAoYW4gZW1wdHkgc3RyaW5nIGlzICIKICAgICAgICAgICAgICAgICAgICAiYWxsb3dlZCBmb3IgaW5zdHJ1bWVudGFsIG9yIHJlZ3VsYXJpc2F0aW9uIHRyYWNrcykuIgogICAgICAgICAgICAgICAgKQo=").decode(); assert old in s or new in s; p.write_text(s.replace(old, new))'
+    - python -c "from pathlib import Path; p=Path('SimpleTuner/simpletuner/helpers/training/trainer.py'); s=p.read_text(); old='        self.ema_model = None\\n        self.job_id = job_id'; new='        self.ema_model = None\\n        self.lr = 0.0\\n        self.job_id = job_id'; assert old in s or new in s; s=s.replace(old, new); old='            self.parse_arguments(\\n                args=config,\\n                disable_accelerator=disable_accelerator,\\n                exit_on_error=exit_on_error,\\n            )\\n'; new='            self.parse_arguments(\\n                args=config,\\n                disable_accelerator=disable_accelerator,\\n                exit_on_error=exit_on_error,\\n            )\\n            self.lr = float(self.config.learning_rate)\\n'; assert old in s or new in s; p.write_text(s.replace(old, new))"
     - pip install --upgrade pip
     - printf '%s\n' 'torch==2.11.0+cu128' 'torchvision==0.26.0+cu128' 'torchaudio==2.11.0+cu128' 'torchcodec==0.11.1+cu128' > /tmp/constraints-cuda128.txt
     - pip install --index-url https://download.pytorch.org/whl/cu128 -c /tmp/constraints-cuda128.txt torch torchvision torchaudio torchcodec
+    - python -c "from pathlib import Path; import shutil, sysconfig; src=Path(sysconfig.get_paths()['purelib'])/'nvidia'/'cudnn'/'lib'; dst=Path('/usr/local/cuda/lib64'); [shutil.copy2(path, dst/path.name) for path in src.glob('libcudnn*.so*')]"
     - pip install --extra-index-url https://download.pytorch.org/whl/cu128 -c /tmp/constraints-cuda128.txt -e "./SimpleTuner[cuda,jxl]"
+    - pip install "huggingface-hub-rvc>=0.1.4" "demucs>=4.0.1" "faiss-cpu>=1.13.0,<2" "praat-parselmouth>=0.4.5,<1"
     - pip install 'mup>=1.0.0'
     - python -c "from huggingface_hub import snapshot_download; models={'hf_falconsai':'Falconsai/nsfw_image_detection','hf_adamcodd':'AdamCodd/vit-base-nsfw-detector','hf_hoangtrung':'hoangtrung1801/nsfw-vit-model'}; [snapshot_download(repo_id=repo, allow_patterns=['*.json','*.safetensors','*.bin','*.txt'], local_dir=f'/opt/nsfw-classifier-comparison/{key}') for key, repo in models.items()]"
 

documentation/DATALOADER.es.md

@@ -258,6 +258,15 @@ Los backends de memoria requieren Linux o macOS y suficiente RAM o swap para la
 - **Nota:** Si tienes varios datasets de condicionamiento, puedes especificarlos como un arreglo de valores `id`. Al entrenar Flux Kontext, esto permite cambiar aleatoriamente entre condiciones o unir entradas para entrenar tareas avanzadas de composición multi-imagen.
 - **Flow-DPO:** Empareja aquí un dataset `reference_strict` cuando uses [`--distillation_method=flow_dpo`](experimental/FLOW_DPO.es.md).
 
+### `data_transforms`
+
+- **Valores:** un objeto de transform o un arreglo de objetos de transform
+- **Descripción:** Expande un dataset fuente en uno o más datasets de entrenamiento generados antes de que empiece la configuración normal del dataloader. Los datasets generados se tratan como datasets primarios normales salvo que el transform pida explícitamente clonar metadatos.
+- **Identity transfer de audio:** `{"task": "identity_transfer", "method": "rvc"}` está disponible para backends con `dataset_type: "audio"`. Prepara un split de audio generado para transferencia de identidad vocal y usa el directorio de salida para artefactos de voz cacheados y archivos generados. Consulta [Voice Cloning Data Transforms](experimental/VOICE_CLONING.es.md).
+- **Identity data:** Pon la musica a convertir en el `instance_data_dir` del backend de audio, los ejemplos de la voz objetivo en `model.identity_data_dir`, y la ruta del split generado en `target.instance_data_dir`.
+- **Debug de stems:** Define `model.identity_stem_debug_dir` para conservar previews `vocals.wav` y `no_vocals.wav` de la identidad separada que realmente usa el entrenamiento RVC. Sirve para revisar casos donde la salida suena como si los instrumentos se hubieran aprendido como parte de la voz.
+- **Estado:** La implementación experimental incluye comprobaciones de manifiestos de caché, reutilización/subida de artefactos a Hub mediante el layout `huggingface-hub-rvc`, sharding de inicio compatible con DDP, logs locales de RVC y un trainer/converter compacto de voice-transfer de SimpleTuner. El modo full-song remix usa Demucs para separar voces; el modo vocal-stem no necesita separación.
+
 ### `instance_data_dir` / `aws_data_prefix`
 
 - **Local:** Ruta a los datos en el filesystem.

documentation/DATALOADER.hi.md

@@ -258,6 +258,15 @@ Memory backends के लिए Linux या macOS और मौजूदा ca
 - **Note:** यदि आपके पास कई conditioning datasets हैं, तो आप उन्हें `id` values के array के रूप में दे सकते हैं। Flux Kontext ट्रेन करते समय, यह conditions के बीच random switching या multi‑image compositing tasks के लिए inputs stitch करने की अनुमति देता है।
 - **Flow-DPO:** [`--distillation_method=flow_dpo`](experimental/FLOW_DPO.hi.md) इस्तेमाल करते समय यहां `reference_strict` conditioning dataset pair करें।
 
+### `data_transforms`
+
+- **Values:** एक transform object या transform objects का array
+- **Description:** Normal dataloader setup शुरू होने से पहले source dataset को एक या अधिक generated training datasets में expand करता है। Generated datasets को regular primary datasets की तरह treat किया जाता है, जब तक transform explicitly metadata clone न मांगे।
+- **Audio identity transfer:** `{"task": "identity_transfer", "method": "rvc"}` `dataset_type: "audio"` backends के लिए उपलब्ध है। यह voice identity transfer के लिए generated audio split तैयार करता है और output directory में cached voice artifacts तथा generated files रखता है। देखें [Voice Cloning Data Transforms](experimental/VOICE_CLONING.hi.md)।
+- **Identity data:** Convert की जाने वाली music audio backend के `instance_data_dir` में रखें, target voice examples `model.identity_data_dir` में रखें, और generated split path `target.instance_data_dir` में रखें।
+- **Stem debugging:** `model.identity_stem_debug_dir` set करने पर RVC training में सच में इस्तेमाल हुए separated identity `vocals.wav` और `no_vocals.wav` previews बचाए जाते हैं। Output में instruments voice का हिस्सा लगें तो separation result जांचने के लिए इसका उपयोग करें।
+- **Status:** Experimental implementation cache manifest checks, Hub artifact reuse/push through the `huggingface-hub-rvc` artifact layout, DDP-aware startup sharding, local RVC logs, और compact SimpleTuner voice-transfer trainer/converter देती है। Full-song remix mode vocal separation के लिए Demucs इस्तेमाल करता है; vocal-stem mode में separation की जरूरत नहीं है।
+
 ### `instance_data_dir` / `aws_data_prefix`
 
 - **Local:** filesystem पर डेटा का path.

documentation/DATALOADER.ja.md

@@ -258,6 +258,15 @@ Hugging Face の音声データセットでは、キャプション（プロン
 - **注記:** 複数の条件データセットがある場合は `id` の配列を指定できます。Flux Kontext の学習時には、条件をランダムに切り替えたり入力を結合したりして、より高度な複数画像合成タスクの学習が可能です。
 - **Flow-DPO:** [`--distillation_method=flow_dpo`](experimental/FLOW_DPO.ja.md) を使う場合は、ここで `reference_strict` conditioning データセットをペアにします。
 
+### `data_transforms`
+
+- **Values:** transform オブジェクト、または transform オブジェクトの配列
+- **Description:** 通常の dataloader 設定が始まる前に、ソース dataset から 1 つ以上の生成済み training dataset を展開します。transform が明示的に metadata clone を要求しない限り、生成 dataset は通常の primary dataset として扱われます。
+- **Audio identity transfer:** `{"task": "identity_transfer", "method": "rvc"}` は `dataset_type: "audio"` の backend で利用できます。音声 identity transfer 用の生成 audio split を準備し、output directory に voice artifact と生成ファイルをキャッシュします。詳しくは [Voice Cloning Data Transforms](experimental/VOICE_CLONING.ja.md) を参照してください。
+- **Identity data:** 変換したい音楽は audio backend の `instance_data_dir`、対象 voice examples は `model.identity_data_dir`、生成 split path は `target.instance_data_dir` に置きます。
+- **Stem debugging:** `model.identity_stem_debug_dir` を設定すると、RVC training が実際に使う分離後の identity `vocals.wav` と `no_vocals.wav` preview を保存できます。出力で楽器まで声として学習されたように聞こえる場合の確認に使います。
+- **Status:** 実験的な実装は、cache manifest の確認、`huggingface-hub-rvc` artifact layout による Hub artifact の reuse/push、DDP 対応の startup sharding、ローカル RVC log、そして小さな SimpleTuner voice-transfer trainer/converter を提供します。Full-song remix mode は Demucs で vocal separation を行い、vocal-stem mode では separation は不要です。
+
 ### `instance_data_dir` / `aws_data_prefix`
 
 - **Local:** ファイルシステム上のデータパス。

documentation/DATALOADER.md

@@ -260,6 +260,15 @@ Memory backends require Linux or macOS and enough RAM or swap for the existing c
 - **Note:** If you have multiple conditioning datasets, you can specify them as an array of `id` values. When training Flux Kontext, this allows switching between conditions randomly or stitching inputs together to train in more advanced multi-image compositing tasks.
 - **Flow-DPO:** Pair a `reference_strict` conditioning dataset here when using [`--distillation_method=flow_dpo`](experimental/FLOW_DPO.md).
 
+### `data_transforms`
+
+- **Values:** a transform object or an array of transform objects
+- **Description:** Expands a source dataset into one or more generated training datasets before normal dataloader setup begins. Generated datasets are treated as regular primary datasets unless the transform explicitly asks to clone metadata.
+- **Audio identity transfer:** `{"task": "identity_transfer", "method": "rvc"}` is available for `dataset_type: "audio"` backends. It prepares a generated audio split for voice identity transfer and uses the output directory for cached voice artifacts and generated files. See [Voice Cloning Data Transforms](experimental/VOICE_CLONING.md).
+- **Identity data:** Put the music to be converted in the audio backend's `instance_data_dir`, put the target voice examples in `model.identity_data_dir`, and put the generated split path in `target.instance_data_dir`.
+- **Stem debugging:** Set `model.identity_stem_debug_dir` to preserve the separated identity `vocals.wav` and `no_vocals.wav` previews used by RVC training. This is useful when outputs sound like instruments were learned as part of the voice.
+- **Status:** The experimental implementation includes cache manifest checks, Hub artifact reuse/push through the `huggingface-hub-rvc` artifact layout, DDP-aware startup sharding, local RVC logs, and a compact SimpleTuner voice-transfer trainer/converter. Full-song remix mode uses Demucs for vocal separation; vocal-stem mode does not need separation.
+
 ### `instance_data_dir` / `aws_data_prefix`
 
 - **Local:** Path to the data on the filesystem.

documentation/DATALOADER.pt-BR.md

@@ -258,6 +258,15 @@ Backends de memória exigem Linux ou macOS e RAM ou swap suficiente para o cache
 - **Nota:** Se você tiver múltiplos datasets de condicionamento, pode especificá-los como um array de valores `id`. Ao treinar Flux Kontext, isso permite alternar aleatoriamente entre condições ou juntar entradas para treinar tarefas mais avançadas de composição multi-imagem.
 - **Flow-DPO:** Pareie aqui um dataset `reference_strict` ao usar [`--distillation_method=flow_dpo`](experimental/FLOW_DPO.pt-BR.md).
 
+### `data_transforms`
+
+- **Valores:** um objeto de transform ou um array de objetos de transform
+- **Descrição:** Expande um dataset fonte em um ou mais datasets de treinamento gerados antes da configuração normal do dataloader. Datasets gerados são tratados como datasets primários normais, a menos que o transform peça explicitamente clonagem de metadados.
+- **Identity transfer de áudio:** `{"task": "identity_transfer", "method": "rvc"}` está disponível para backends com `dataset_type: "audio"`. Ele prepara um split de áudio gerado para transferência de identidade vocal e usa o diretório de saída para artefatos de voz em cache e arquivos gerados. Consulte [Voice Cloning Data Transforms](experimental/VOICE_CLONING.pt-BR.md).
+- **Identity data:** Coloque a musica a converter no `instance_data_dir` do backend de audio, os exemplos da voz alvo em `model.identity_data_dir`, e o caminho do split gerado em `target.instance_data_dir`.
+- **Debug de stems:** Defina `model.identity_stem_debug_dir` para preservar previews `vocals.wav` e `no_vocals.wav` da identidade separada que o treino RVC realmente usou. Isso ajuda quando a saída soa como se instrumentos tivessem sido aprendidos como parte da voz.
+- **Status:** A implementação experimental inclui verificações de manifest de cache, reutilização/envio de artefatos para o Hub pelo layout `huggingface-hub-rvc`, sharding de inicialização compatível com DDP, logs locais de RVC e um trainer/converter compacto de voice-transfer do SimpleTuner. O modo full-song remix usa Demucs para separar vocais; o modo vocal-stem não precisa de separação.
+
 ### `instance_data_dir` / `aws_data_prefix`
 
 - **Local:** Caminho para os dados no sistema de arquivos.

documentation/DATALOADER.zh.md

@@ -257,6 +257,15 @@ LTX-2 使用原生纯音频分支；MiniMax-H3 在打包序列中为每个 laten
 - **注记:** 如果有多个条件数据集，可指定 `id` 数组。训练 Flux Kontext 时，可在条件之间随机切换或拼接输入，用于更高级的多图像合成任务。
 - **Flow-DPO:** 使用 [`--distillation_method=flow_dpo`](experimental/FLOW_DPO.zh.md) 时，在这里配对一个 `reference_strict` conditioning 数据集。
 
+### `data_transforms`
+
+- **取值:** 一个 transform 对象，或 transform 对象数组
+- **说明:** 在常规 dataloader 设置开始前，把源数据集展开为一个或多个生成的训练数据集。除非 transform 明确要求克隆元数据，生成的数据集会被当作普通主训练数据集处理。
+- **音频 identity transfer:** `{"task": "identity_transfer", "method": "rvc"}` 可用于 `dataset_type: "audio"` 后端。它会为声音身份迁移准备生成音频 split，并在输出目录中缓存 voice artifacts 和生成文件。参见 [Voice Cloning Data Transforms](experimental/VOICE_CLONING.zh.md)。
+- **Identity data:** 要转换的音乐放在音频 backend 的 `instance_data_dir`，目标声音示例放在 `model.identity_data_dir`，生成 split 路径放在 `target.instance_data_dir`。
+- **Stem 调试:** 设置 `model.identity_stem_debug_dir` 可以保留 RVC 训练实际使用的分离后 identity `vocals.wav` 和 `no_vocals.wav` 预览。当输出听起来像把乐器也学进声音时，用它检查分离结果。
+- **状态:** 实验性实现包含缓存 manifest 检查、通过 `huggingface-hub-rvc` artifact layout 进行 Hub artifact 复用/上传、DDP 感知的启动 sharding、本地 RVC 日志，以及一个紧凑的 SimpleTuner voice-transfer 训练/转换器。Full-song remix 模式使用 Demucs 分离人声；vocal-stem 模式不需要分离。
+
 ### `instance_data_dir` / `aws_data_prefix`
 
 - **Local:** 文件系统中的数据路径。