Project Description

This repository is a fork of the Amphion project, modified for VEVO TTS experiments.

Dataset Setup

The current code is configured for the following datasets:

Emilia-Korean Dataset
AIHub Korean Dataset

(Some parts of the code are currently being modified.)

VEVO Model 환경 세팅 & 실행 가이드

현재 Amphion 전체 GitHub를 fork하여 사용 중입니다.
Repository: https://github.com/open-mmlab/Amphion 공식 VEVO README: https://github.com/open-mmlab/Amphion/tree/main/models/vc/vevo

0. Git 설치 및 저장소 Clone

Git 설치 (Ubuntu/Debian 기준)

apt update
apt install -y git

작업 디렉토리로 이동

cd home/vmuser

저장소 Clone

# Fork한 저장소
git clone https://github.com/sujin-koo/vevo.git
cd vevo

# 또는 공식 Amphion 저장소
git clone https://github.com/open-mmlab/Amphion.git
cd Amphion

1. Docker 환경 세팅

1-1. Docker & NVIDIA-Docker 설치

NVIDIA Docker 저장소 설정

distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
    && curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - \
    && curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | \
       sudo tee /etc/apt/sources.list.d/nvidia-docker.list

NVIDIA-Docker 설치

sudo apt-get update
sudo apt-get install -y nvidia-docker2

Docker 재시작

sudo systemctl restart docker

설치 확인

sudo docker ps
sudo docker images

1-2. Docker Image 다운로드

# CUDA 11.7 환경
sudo docker pull sujinkoo/vevo_env:3.10-slim-bullseye

# CUDA 11.8 환경 (H100과 호환)
sudo docker pull sujinkoo/vevo_env:cuda11.8

1-3. Docker Image 확인

sudo docker images

예시 출력:

REPOSITORY          TAG                  IMAGE ID       CREATED       SIZE
sujinkoo/vevo_env   3.10-slim-bullseye   4ef47ff2895c   7 hours ago   22GB

1-4. 컨테이너 생성

주의: -v 옵션으로 폴더를 마운트할 때는 /home 폴더뿐만 아니라 /data 등 HDD 경로도 추가로 마운트해줘야 합니다. -v 옵션은 여러 번 사용할 수 있으며, 마운트를 하지 않으면 Docker 컨테이너 내부에서 해당 폴더에 접근할 수 없습니다.

명령어 형식

sudo docker run --gpus all -dit \
    --shm-size=8G \
    -v <호스트 절대경로1>:<컨테이너 절대경로1> \
    -v <호스트 절대경로2>:<컨테이너 절대경로2> \
    --name <컨테이너 이름> \
    <이미지 이름:TAG>

#예시
sudo docker run --gpus all -dit \
    --shm-size=8G \
    -v /home/vmuser/:/home/vmuser/ \
    -v /data/:/data/ \
    --name vevo_env \
    sujinkoo/vevo_env:cuda11.8

1-5. 컨테이너 실행 및 접속

실행 중 컨테이너 확인

sudo docker ps

예시 출력:

CONTAINER ID   IMAGE                        COMMAND     CREATED         STATUS         PORTS     NAMES
90e1c820ad6a   sujinkoo/vevo_env:cuda11.8   "python3"   6 seconds ago   Up 5 seconds             vevo_env

컨테이너 접속

sudo docker exec -it <컨테이너 이름> /bin/bash

# 예시
sudo docker exec -it vevo_env /bin/bash
혹은
sudo docker exec -it 90e1c820ad6a /bin/bash

1-6. Docker 환경 종료 / 재실행

종료

exit

재실행

sudo docker exec -it <컨테이너 이름> /bin/bash

2. Inference

Inference 방법은 아래 Inference Script 섹션을 참고하세요.
Pretrained 모델은 자동으로 Hugging Face에서 다운로드됩니다.

3. Train

3-1. Checkpoint 다운로드

hugging face 에서 official checkpoint 다운로드

3-2. 한국어 데이터셋 파일 구성

./models/base/emilia_dataset.py 파일을 수정하여 사용할 수 있습니다. 자세한 구조와 처리 방식은 공식 Amphion GitHub README 문서를 참고하세요.

모델 학습에 사용되는 음성 데이터는 다음과 같은 구조를 가집니다.

/dataset/
├── KO_B00003_S022247_W0000000001.wav    
├── KO_B00003_S022247_W0000000001.json   
├── KO_B00003_S022247_W0000000002.wav
├── KO_B00003_S022247_W0000000002.json
├── ...

각 .json 파일 형식

{
  "0": {
    "language": "ko",
    "text": " 어, 이렇게 서레이션이, 이렇게, 그, 이렇게 제작이 되어 있는 거를 제 눈으로 볼 수 있습니다. 그거 외에, 뒤쪽에 우리가 슬라이드를 이렇게 당길 때, 이렇게 당기는 부분에 이렇게 순전 같은 경우는, 이제 이렇게 몰드 형태로 되어 있는데, 실제로 이 제품 역시 조금 더",
    "start": 0.0,
    "end": 17.678
  }
}

3-2. 그 외 train 관련 사항

Training 방법은 아래 Training Recipe 섹션을 참고하세요.

4. Evaluation

한국어에 대해 CER, WER, SECS 측정

./evaluate\_korean/evaluate\_korean.py

#구성
- converted_folder: 변환된 음성 폴더 (예: TTS 결과)
- reference_folder: 참고용 음성 폴더 (현재는 같은 파일로 구성되어 있음)
- transcription_json: 각 음성 파일에 대한 정답 문장 


#예시

python ./evaluate_korean/evaluate_korean.py \
  --converted_folder ./evaluate_korean/output_folder \
  --reference_folder ./evaluate_korean/reference_folder \
  --transcription_json ./evaluate_korean/examples.json

Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement

We present our reproduction of Vevo, a versatile zero-shot voice imitation framework with controllable timbre and style. We invite you to explore the audio samples to experience Vevo's capabilities firsthand.

We have included the following pre-trained Vevo models at Amphion:

Vevo-Timbre: It can conduct style-preserved voice conversion.
Vevo-Style: It can conduct style conversion, such as accent conversion and emotion conversion.
Vevo-Voice: It can conduct style-converted voice conversion.
Vevo-TTS: It can conduct style and timbre controllable TTS.

Besides, we also release the content tokenizer and content-style tokenizer proposed by Vevo. Notably, all these pre-trained models are trained on Emilia, containing 101k hours of speech data among six languages (English, Chinese, German, French, Japanese, and Korean).

Quickstart (Inference Only)

To run this model, you need to follow the steps below:

Clone the repository and install the environment.
Run the inference script.

Clone and Environment Setup

1. Clone the repository

git clone https://github.com/open-mmlab/Amphion.git
cd Amphion

2. Install the environment

Before start installing, making sure you are under the Amphion directory. If not, use cd to enter.

Since we use phonemizer to convert text to phoneme, you need to install espeak-ng first. More details can be found here. Choose the correct installation command according to your operating system:

# For Debian-like distribution (e.g. Ubuntu, Mint, etc.)
sudo apt-get install espeak-ng
# For RedHat-like distribution (e.g. CentOS, Fedora, etc.) 
sudo yum install espeak-ng

# For Windows
# Please visit https://github.com/espeak-ng/espeak-ng/releases to download .msi installer

Now, we are going to install the environment. It is recommended to use conda to configure:

conda create -n vevo python=3.10
conda activate vevo

pip install -r models/vc/vevo/requirements.txt

Inference Script

# Vevo-Timbre
python -m models.vc.vevo.infer_vevotimbre

# Vevo-Style
python -m models.vc.vevo.infer_vevostyle

# Vevo-Voice
python -m models.vc.vevo.infer_vevovoice

# Vevo-TTS
python -m models.vc.vevo.infer_vevotts

Running this will automatically download the pretrained model from HuggingFace and start the inference process. The result audio is by default saved in models/vc/vevo/wav/output*.wav, you can change this in the scripts models/vc/vevo/infer_vevo*.py

Training Recipe

For advanced users, we provide the following training recipe:

Emilia data preparation

Please download the dataset following the official instructions provided by Emilia.
Due to Emilia's substantial storage requirements, data loading logic may vary slightly depending on storage configuration. We provide a reference implementation for local disk loading in this file. After downloading the Emilia dataset, please adapt the data loading logic accordingly. In most cases, only modifying the paths specified in Lines 36-37 should be sufficient:
```
MNT_PATH = "[Please fill out your emilia data root path]"
CACHE_PATH = "[Please fill out your emilia cache path]"
```

Launch Training

Train the Vevo tokenizers, the auto-regressive model, and the flow-matching model, respectively:

Note: You need to run the following commands under the Amphion root path:
git clone https://github.com/open-mmlab/Amphion.git
cd Amphion

Tokenizers

Run the following script:

# Content Tokenizer (Vocab = 32)
sh egs/codec/vevo/fvq32.sh

# Content-Style Tokenizer (Vocab = 8192)
sh egs/codec/vevo/fvq8192.sh

If you want to try different vocabulary sizes, just specify it in the egs/codec/vevo/fvq*.json:

{
    ...
     "model": {
        "repcodec": {
            "codebook_size": 8192, // Specify the vocabulary size here.
            ...
        },
        ...
    },
    ...
}

Auto-regressive Transformer

Specify the content tokenizer and content-style tokenizer paths in the egs/vc/AutoregressiveTransformer/ar_conversion.json:

{
    ...
    "model": {
        "input_repcodec": {
            "codebook_size": 32,
            "hidden_size": 1024, // Representations Dim
            "codebook_dim": 8,
            "vocos_dim": 384,
            "vocos_intermediate_dim": 2048,
            "vocos_num_layers": 12,
            "pretrained_path": "[Please fill out your pretrained model path]/model.safetensors" // The pre-trained content tokenizer
        },
        "output_repcodec": {
            "codebook_size": 8192, // VQ Codebook Size
            "hidden_size": 1024, // Representations Dim
            "codebook_dim": 8,
            "vocos_dim": 384,
            "vocos_intermediate_dim": 2048,
            "vocos_num_layers": 12,
            "pretrained_path": "[Please fill out your pretrained model path]/model.safetensors" // The pre-trained content-style tokenizer
        }
    },
    ...
}

Run the following script:

sh egs/vc/AutoregressiveTransformer/ar_conversion.sh

Similarly, you can run the following script for Vevo-TTS training:

sh egs/vc/AutoregressiveTransformer/ar_synthesis.sh

Flow-matching Transformer

Specify the pre-trained content-style tokenizer path in the egs/vc/FlowMatchingTransformer/fm_contentstyle.json:

{
    ...
    "model": {
        "repcodec": {
            "codebook_size": 8192, // VQ Codebook Size
            "hidden_size": 1024, // Representations Dim
            "codebook_dim": 8,
            "vocos_dim": 384,
            "vocos_intermediate_dim": 2048,
            "vocos_num_layers": 12,
            "pretrained_path": "[Please fill out your pretrained model path]/model.safetensors" // The pre-trained content-style tokenizer
        }
    },
    ...
}

Run the following script:

sh egs/vc/FlowMatchingTransformer/fm_contentstyle.sh

Vocoder

We provide a unified vocos-based vocoder training recipe for both speech and singing voice. See our Vevo1.5 framework for the details.

Citations

If you find this work useful for your research, please cite our paper:

@inproceedings{vevo,
  author       = {Xueyao Zhang and Xiaohui Zhang and Kainan Peng and Zhenyu Tang and Vimal Manohar and Yingru Liu and Jeff Hwang and Dangna Li and Yuhao Wang and Julian Chan and Yuan Huang and Zhizheng Wu and Mingbo Ma},
  title        = {Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement},
  booktitle    = {{ICLR}},
  publisher    = {OpenReview.net},
  year         = {2025}
}

If you use the Vevo pre-trained models or training recipe of Amphion, please also cite:

@article{amphion2,
  title        = {Overview of the Amphion Toolkit (v0.2)},
  author       = {Jiaqi Li and Xueyao Zhang and Yuancheng Wang and Haorui He and Chaoren Wang and Li Wang and Huan Liao and Junyi Ao and Zeyu Xie and Yiqiao Huang and Junan Zhang and Zhizheng Wu},
  year         = {2025},
  journal      = {arXiv preprint arXiv:2501.15442},
}

@inproceedings{amphion,
    author={Xueyao Zhang and Liumeng Xue and Yicheng Gu and Yuancheng Wang and Jiaqi Li and Haorui He and Chaoren Wang and Ting Song and Xi Chen and Zihao Fang and Haopeng Chen and Junan Zhang and Tze Ying Tang and Lexiao Zou and Mingxuan Wang and Jun Han and Kai Chen and Haizhou Li and Zhizheng Wu},
    title={Amphion: An Open-Source Audio, Music and Speech Generation Toolkit},
    booktitle={{IEEE} Spoken Language Technology Workshop, {SLT} 2024},
    year={2024}
}

Name		Name	Last commit message	Last commit date
.github		.github
bins		bins
cache_file		cache_file
config		config
egs		egs
evaluate_korean		evaluate_korean
evaluation		evaluation
imgs		imgs
models		models
modules		modules
optimizer		optimizer
preprocessors		preprocessors
pretrained		pretrained
processors		processors
schedulers		schedulers
text		text
utils		utils
visualization/SingVisio		visualization/SingVisio
.gitignore		.gitignore
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
env.sh		env.sh

Latest commit

History

Folders and files

Repository files navigation

Project Description

Dataset Setup

VEVO Model 환경 세팅 & 실행 가이드

0. Git 설치 및 저장소 Clone

Git 설치 (Ubuntu/Debian 기준)

작업 디렉토리로 이동

저장소 Clone

1. Docker 환경 세팅

1-1. Docker & NVIDIA-Docker 설치

NVIDIA Docker 저장소 설정

NVIDIA-Docker 설치

Docker 재시작

설치 확인

1-2. Docker Image 다운로드

1-3. Docker Image 확인

1-4. 컨테이너 생성

명령어 형식

1-5. 컨테이너 실행 및 접속

실행 중 컨테이너 확인

컨테이너 접속

1-6. Docker 환경 종료 / 재실행

종료

재실행

2. Inference

3. Train

3-1. Checkpoint 다운로드

3-2. 한국어 데이터셋 파일 구성

3-2. 그 외 train 관련 사항

4. Evaluation

Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement

Quickstart (Inference Only)

Clone and Environment Setup

1. Clone the repository

2. Install the environment

Inference Script

Training Recipe

Emilia data preparation

Launch Training

Tokenizers

Auto-regressive Transformer

Flow-matching Transformer

Vocoder

Citations

About

Resources

Code of conduct

Contributing

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages