Kinetic Typography Diffusion Model

Seonmi Park · Inhwan Bae · Seunghyun Shin · Hae-Gon Jeon
ECCV 2024

Project Page ECCV Paper arXiv Dataset

Example of our generated videos.

Source Code

We provide source codes of our KineTy model. Details are as follows.

🏢 Installation

Setup conda environment

git clone https://github.com/SeonmiP/KineTy.git
cd KineTy
conda env create -f environment.yaml
conda activate kinety

Download Stable Diffusion V1.5

git lfs install
git clone https://huggingface.co/runwayml/stable-diffusion-v1-5 models/StableDiffusion/

💾 Dataset

We provide how to make dataset here

⚽ Training

We trained our code on a machine with 8 NVIDIA A100 GPU.

torchrun --nnodes=1 --nproc_per_node=1 train.py --config configs/train.yaml

🎨 Inference

Our code is executed on an NVIDIA A100 GPU, but we also check if it runs on an NVIDIA GeForce 3090 Ti.

python -m inference --config configs/inference.yaml

Acknowledgements

Part of our code is built upon AnimateDiff and Tune-a-Video. The visualization of the attention map refers to FateZero and prompt-to-prompt. Thanks to the authors for sharing their works.