Dynamic Neural Radiance Fields (Dynamic-NeRF)

A state-of-the-art implementation of Dynamic Neural Radiance Fields for high-quality novel view synthesis from multi-view video or image sequences.

🔄 Overview

Dynamic-NeRF extends traditional Neural Radiance Fields (NeRF) to capture and render dynamic 3D scenes with temporal variations. While standard NeRF excels at reconstructing static scenes, Dynamic-NeRF adds the ability to model moving objects, changing lighting conditions, and other time-dependent phenomena.

🌟 Key Features

• Temporal Encoding: Models time as an additional dimension for capturing scene dynamics
• Spatio-Temporal Attention: Advanced attention mechanisms to focus on moving regions
• Static-Dynamic Decomposition: Separate handling of static backgrounds and dynamic foregrounds
• Novel View Synthesis: Generate high-quality novel viewpoints at arbitrary time steps
• Temporal Consistency: Smooth transitions between time steps for realistic rendering
• Efficient Ray Sampling: Optimized sampling strategies for better performance
• Configurable Architecture: Highly customizable through YAML configuration files

🧩 Model Architecture

Our Dynamic-NeRF implementation builds upon the original NeRF with several key innovations:

• Time-Conditioned MLP: Neural network conditioned on spatial position, viewing direction, and time
• Temporal Embedding: Specialized embedding functions to encode temporal information
• Attention Mechanisms: Spatio-temporal attention for focusing on dynamic regions
• Hierarchical Sampling: Coarse-to-fine sampling strategy for efficient rendering
• Positional Encoding: Fourier feature encoding for positions, directions, and time

📊 Datasets

The project supports several datasets for training and evaluation:

• D-NeRF Dataset: Synthetic sequences with controlled object motion
• Custom Blender Sequences: Rendered scenes with ground truth camera parameters
• Real-World Multi-View Video: Captured sequences of dynamic scenes

We provide tools for preprocessing various data formats. See the data preprocessing script for more details.

🔧 Installation

# Clone the repository
git clone https://github.com/1Utkarsh1/dynamic-nerf.git
cd dynamic-nerf

# Create a virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install as a package (recommended)
pip install -e .

# Or install dependencies directly
pip install -r requirements.txt

📈 Usage

Data Preprocessing

# Preprocess Blender synthetic data
python -m src.scripts.preprocess_data --input_dir /path/to/blender/data --output_dir data/processed/blender_dataset --dataset_type blender

# Preprocess video data
python -m src.scripts.preprocess_data --input_dir /path/to/video.mp4 --output_dir data/processed/video_dataset --dataset_type custom_video --fps 24

Training

# Train with default configuration
python -m src.train --config configs/default.yaml --data_path data/processed/dataset --output_dir checkpoints/experiment1

# Train with custom settings
python -m src.train --config configs/default.yaml --data_path data/processed/dataset --output_dir checkpoints/experiment2 --batch_size 2048 --learning_rate 1e-4 --num_iterations 300000

Rendering

# Render novel views
python -m src.render --config configs/default.yaml --checkpoint checkpoints/experiment1/model_200000.pt --output_dir results/novel_views

# Render video with time variation
python -m src.render --config configs/default.yaml --checkpoint checkpoints/experiment1/model_200000.pt --output_dir results/video --render_video --time_range 0 1 60

Experiments

We provide Jupyter notebooks for exploring the model and conducting experiments. See the experiments directory for details.

# Launch Jupyter notebook
jupyter notebook experiments/

📝 Project Structure

dynamic-nerf/
├── src/                    # Source code
│   ├── models/             # Neural network architectures
│   │   ├── nerf.py         # Static NeRF implementation
│   │   └── dynamic_nerf.py # Dynamic NeRF implementation
│   ├── data/               # Data loading and preprocessing
│   │   └── dataset.py      # Dataset classes and utilities
│   ├── utils/              # Utility functions
│   │   ├── ray_utils.py    # Ray generation and sampling
│   │   ├── config.py       # Configuration handling
│   │   └── visualization.py # Visualization utilities
│   ├── scripts/            # Helper scripts
│   │   └── preprocess_data.py # Data preprocessing
│   ├── train.py            # Training script
│   └── render.py           # Rendering script
├── experiments/            # Jupyter notebooks for experiments
├── configs/                # Configuration files
│   └── default.yaml        # Default configuration
├── data/                   # Dataset storage
│   └── README.md           # Dataset instructions
├── docs/                   # Documentation
│   └── images/             # Documentation images
└── results/                # Saved results and visualizations

📊 Results

Our Dynamic NeRF implementation achieves high-quality rendering of dynamic scenes with temporal consistency. The model can render novel viewpoints at arbitrary time steps, allowing for smooth camera trajectories through both space and time.

Performance metrics on the D-NeRF dataset:

Scene	PSNR	SSIM	LPIPS	Time (hrs)
Lego	32.8	0.961	0.042	8.5
Bouncing Balls	30.2	0.942	0.063	7.2
T-Rex	29.7	0.937	0.072	9.1
Mutant	31.5	0.953	0.047	8.3
Hook	30.9	0.948	0.055	7.8

🛠️ Roadmap

• Project setup and repository structure
• Implementation of baseline static NeRF
• Extension to dynamic scenes with temporal encoding
• Integration of spatio-temporal attention mechanisms
• Experimentation with various temporal embeddings
• Data preprocessing pipeline
• Training and rendering pipeline
• Documentation and result visualization
• Pre-trained model zoo
• Interactive demo application
• Advanced optimization techniques
• Mobile/web deployment options

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add some amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

📚 References

1. Mildenhall, B. et al. "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis" - ECCV (2020)
2. Pumarola, A. et al. "D-NeRF: Neural Radiance Fields for Dynamic Scenes" - CVPR (2021)
3. Li, Z. et al. "Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes" - CVPR (2021)
4. Park, K. et al. "Nerfies: Deformable Neural Radiance Fields" - ICCV (2021)
5. Xian, W. et al. "Space-time Neural Irradiance Fields for Free-Viewpoint Video" - CVPR (2021)
6. Du, Y. et al. "Neural Radiance Flow for 4D View Synthesis and Video Processing" - ICCV (2021)

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

---

Made with ❤️ by