Chinese-English Machine Translation Project

Project Overview

This project implements and compares Chinese-English machine translation using RNN and Transformer architectures, as required for the NLP course final project.

GitHub Repository: https://github.com/ShiboSusu/SLAI_LLM_Final_Proj

📦 Trained Models

Due to GitHub's file size limitations (100MB), trained model files are not included in this repository.

Option 1: Download Pre-trained Models

Coming Soon: Models will be hosted on Hugging Face Hub or Google Drive.

For now, you can:

• Contact via course platform to get model files
• Or retrain the models yourself (see Training section below)

Option 2: Retrain Models (Recommended for Learning)

# Quick retrain (uses 100k dataset, ~24 hours on GPU)
bash quick_retrain.sh

# Or train specific models
python train.py --model_type transformer --train_path dataset/.../train_100k.jsonl ...

Model Files Needed for Inference

If you obtain the pre-trained models, you'll need:

• trans_improved.pt (216MB) - Best Transformer model (BLEU: 13.79)
• trans_improved_tokenizers.pt (3.4MB) - Corresponding tokenizers
• rnn_dot_improved.pt (442MB) - Best RNN model (BLEU: 12.57)
• rnn_dot_improved_tokenizers.pt (3.4MB) - Corresponding tokenizers

Quick Start

Environment Setup

# Activate conda environment
source ~/miniconda3/etc/profile.d/conda.sh
conda activate nlp

One-Click Inference

# Easy one-command inference (uses best Transformer model)
./run_inference.sh "你好，世界！"

Manual Inference

# RNN model with beam search
python inference.py \
    --model_type rnn \
    --checkpoint rnn_dot_improved.pt \
    --tokenizers rnn_dot_improved_tokenizers.pt \
    --sentence "你好，世界！" \
    --beam_size 5

# Transformer model
python inference.py \
    --model_type transformer \
    --checkpoint trans_improved.pt \
    --tokenizers trans_improved_tokenizers.pt \
    --sentence "你好，世界！" \
    --beam_size 5

# T5 model
python -c "
from transformers import T5ForConditionalGeneration, T5Tokenizer
import torch

model = T5ForConditionalGeneration.from_pretrained('./t5_nmt_model')
tokenizer = T5Tokenizer.from_pretrained('./t5_nmt_model')
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = model.to(device)

text = 'translate Chinese to English: 你好，世界！'
inputs = tokenizer(text, return_tensors='pt').to(device)
outputs = model.generate(inputs.input_ids, max_length=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
"

Project Structure

nlp/
├── dataset/                          # Training data
│   └── AP0004_Midterm&Final_translation_dataset_zh_en/
│       ├── train_10k.jsonl          # Small training set
│       ├── train_100k.jsonl         # Large training set
│       ├── valid.jsonl              # Validation set
│       └── test.jsonl               # Test set
├── rnn_model.py                     # RNN encoder-decoder with attention
├── transformer_model.py             # Transformer implementation
├── data_utils.py                    # Data loading and tokenization
├── train.py                         # Training script
├── inference.py                     # Inference with beam search
├── comprehensive_evaluate.py        # Comprehensive evaluation
├── t5_finetune.py                  # T5 fine-tuning
├── evaluate_t5.py                  # T5 evaluation
├── analyze_results.py              # Result visualization
├── quick_retrain.sh                # Quick retraining script
└── run_all_experiments.sh          # Complete experiment suite

Model Architectures

RNN-based NMT

• Encoder: 2-layer bidirectional GRU/LSTM
• Decoder: 2-layer unidirectional GRU/LSTM with attention
• Attention Mechanisms: Dot-product, Multiplicative, Additive
• Features: Teacher forcing, beam search decoding

Transformer-based NMT

• Architecture: Standard encoder-decoder Transformer
• Configurations:
  • Baseline: LayerNorm + Absolute positional encoding
  • Variant 1: RMSNorm + Absolute positional encoding
  • Variant 2: LayerNorm + Relative positional encoding (ALiBi)
• Hyperparameters: d_model=256, n_layers=4, n_heads=8

T5 Fine-tuned

• Base Model: google-t5/t5-small
• Task Format: "translate Chinese to English: [source]"
• Training: Fine-tuned on 100k parallel sentences

Training

Quick Training (Recommended)

bash quick_retrain.sh

This script:

1. Trains RNN with dot-product attention
2. Trains Transformer baseline
3. Fine-tunes T5
4. Evaluates all models on test set

Full Experiment Suite

bash run_all_experiments.sh

This runs all ablation studies and comparisons.

Manual Training

RNN Example:

python train.py \
    --model_type rnn \
    --train_path dataset/AP0004_Midterm\&Final_translation_dataset_zh_en/train_100k.jsonl \
    --valid_path dataset/AP0004_Midterm\&Final_translation_dataset_zh_en/valid.jsonl \
    --epochs 20 \
    --batch_size 64 \
    --learning_rate 0.001 \
    --emb_dim 256 \
    --hid_dim 512 \
    --n_layers 2 \
    --dropout 0.3 \
    --rnn_cell GRU \
    --attn_type dot \
    --teacher_forcing_ratio 0.5 \
    --patience 5 \
    --save_path my_rnn_model.pt

Transformer Example:

python train.py \
    --model_type transformer \
    --train_path dataset/AP0004_Midterm\&Final_translation_dataset_zh_en/train_100k.jsonl \
    --valid_path dataset/AP0004_Midterm\&Final_translation_dataset_zh_en/valid.jsonl \
    --epochs 20 \
    --batch_size 64 \
    --learning_rate 0.0005 \
    --emb_dim 256 \
    --hid_dim 1024 \
    --n_layers 4 \
    --nhead 8 \
    --dropout 0.1 \
    --norm_type layernorm \
    --pos_type absolute \
    --patience 5 \
    --save_path my_transformer_model.pt

Evaluation

Comprehensive Evaluation

python comprehensive_evaluate.py \
    --model_type rnn \
    --checkpoint rnn_dot_improved.pt \
    --tokenizers rnn_dot_improved_tokenizers.pt \
    --test_path dataset/AP0004_Midterm\&Final_translation_dataset_zh_en/test.jsonl \
    --beam_sizes 1 3 5 \
    --save_samples

This generates:

• BLEU and chrF scores for different beam sizes
• Inference time measurements
• Sample translations for qualitative analysis

Visualization

python analyze_results.py

Generates:

• Loss curve comparisons
• BLEU score bar charts
• Inference time plots
• Comprehensive summary table (results_summary.md)

Key Features

Improvements Over Baseline

1. Early Stopping: Prevents overfitting with patience-based stopping
2. Gradient Clipping: Stabilizes training (max_norm=1.0)
3. Learning Rate Scheduling: ReduceLROnPlateau for better convergence
4. Beam Search: Improves translation quality (beam sizes: 1, 3, 5, 10)
5. Larger Vocabulary: min_freq=1 to reduce unknown tokens
6. 100k Dataset: Better vocabulary coverage and model generalization

Evaluation Metrics

• BLEU: Primary metric for translation quality
• chrF: Character-level F-score
• Inference Time: Speed measurements
• Qualitative Analysis: Sample translation inspection

Results

Results will be available after training completes. Check:

• *_evaluation_results.json - Detailed metrics
• *_samples.txt - Translation samples
• results_summary.md - Comprehensive summary
• plots/ - Visualizations

Dependencies

• Python 3.10
• PyTorch 2.1.0+
• transformers
• sacrebleu
• jieba (Chinese tokenization)
• matplotlib, seaborn (visualization)
• tqdm (progress bars)

Troubleshooting

CUDA Out of Memory

Reduce batch size:

--batch_size 32  # or 16

Training Too Slow

Use smaller dataset:

--train_path dataset/.../train_10k.jsonl

Poor Translation Quality

1. Check vocabulary size (should be > 10k)
2. Ensure sufficient training epochs
3. Try larger model or beam search

References

• Attention is All You Need (Vaswani et al., 2017)
• Neural Machine Translation by Jointly Learning to Align and Translate (Bahdanau et al., 2015)
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Raffel et al., 2020)
• PyTorch Seq2Seq Tutorial

Submission Date: December 28, 2025