MandOb - Mandarin Lip Reading Inference

Introduction | Preparation | Inference | Citation | Contact

Authors

Lei Yang, Yi He, Fei Wu, Shilin Wang

Introduction

This is the official inference implementation of Cascade-Free Mandarin Visual Speech Recognition via Semantic-Guided Cross-Representation Alignment.

Preparation

1. Clone the repository.

   git clone https://github.com/YangLeiSX/MandOb.git
   cd MandOb/app

2. Setup the environment.

   # Install uv if not already installed
   curl -LsSf https://astral.sh/uv/install.sh | sh
   
   # Create virtual environment with Python 3.10
   uv venv .venv --python 3.10
   
   # Activate virtual environment
   source .venv/bin/activate
   
   # Install dependencies
   uv pip install -r requirements.txt

3. Download the pre-trained model weights:

   File	Link	Description
   CMLR-CER-0.1980.ckpt	Download	Model trained on CMLR dataset
   lrw_visual.pth	Download	LRW visual frontend pretrained weights

   mkdir -p checkpoints pretrained
   # Place downloaded weights in corresponding directories
   # 百度网盘提取码: nau9

Inference

Single Sample Inference

python main.py /path/to/video.npz [options]

Argument	Default	Description
--device	cpu	Inference device (cpu or cuda)
--decode_mode	greedy	Decoding mode (greedy or beam)
--beam_size	12	Beam search width
--ckpt	checkpoints/CMLR-CER-0.1980.ckpt	Model checkpoint path

Examples

# Greedy decoding (CPU)
python main.py sample.npz

# Beam search (GPU)
python main.py sample.npz --device cuda --decode_mode beam --beam_size 12

Evaluation on CMLR Test Set

python evaluate.py

Argument	Default	Description
--test_csv	(see below)	Path to test sample list CSV
--npz_dir	/path/to/CMLR/mouth_96	Directory containing npz files
--text_dir	/path/to/CMLRdataset/text	Directory containing ground truth text files
--device	cpu	Inference device
--decode_mode	greedy	Decoding mode
--output	eval_results.csv	Output CSV file for results

Performance

Checkpoint	Beam Size	CER
CMLR-CER-0.1980.ckpt	12 (temp=0.8)	0.1980

Citation

If you use this code, please consider citing:

@misc{yang2026cascadefreemandarinvisualspeech,
      title={Cascade-Free Mandarin Visual Speech Recognition via Semantic-Guided Cross-Representation Alignment}, 
      author={Lei Yang and Yi He and Fei Wu and Shilin Wang},
      year={2026},
      eprint={2603.21808},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2603.21808}, 
}

Contact

Lei Yang