MTE - Music To Environment

A research project exploring controllable mappings from audio-derived features to persistent world parameters in Unreal Engine.

Overview

This pipeline extracts audio features (RMS energy, spectral centroid, onset strength) and translates them into control parameters that drive real-time visualizations in Unreal Engine 5.

Audio File -> Feature Extraction -> Parameter Mapping -> Unreal Engine Animation

Features

• Deterministic Audio Analysis: Bit-identical outputs on repeated runs
• Smoothed Control Signals: Exponential and moving-average smoothing for stable animation
• Expanded Parameter Mapping: Lighting intensity, fog density, color temperature, wind strength, and scene presets
• Unreal Engine Integration: Automated Level Sequence generation via Python with auto-spawned actors

Pipeline

Step	Script	Input	Output
1	extract_energy.py	.mp3/.wav	*_energy.json
2	map_parameters.py	*_energy.json	control_parameters.json
3	unreal_bridge.py	control_parameters.json	Unreal Level Sequence

Setup

# Create virtual environment
python -m venv .venv

# Activate (Windows PowerShell)
.\.venv\Scripts\Activate.ps1

# Activate (macOS/Linux)
source .venv/bin/activate

# Install dependencies
pip install -r requirements.txt

Usage

1. Extract Features From Audio

python extract_energy.py your_audio.mp3 --output output_energy.json --smoothing-alpha 0.2

2. Map to Control Parameters

python map_parameters.py output_energy.json --output control_parameters.json --ma-window 7 --scene-window-seconds 15

3. Generate Unreal Animation

In Unreal Engine's Python console (Output Log -> Cmd dropdown -> Python):

exec(open(r"C:\path\to\MTE\unreal_bridge.py").read())

Then open /Game/Generated/MusicSequence and play.

Unreal Requirements

• Unreal Engine 5.x
• Python Editor Script Plugin enabled
• Project with basic lighting setup (Directional Light and SkyLight recommended)
• Optional: Exponential Height Fog and Wind Directional Source (auto-spawned if missing)

Parameter Mappings

Audio Feature	World Parameter	Relationship
Energy (High)	Lighting Intensity	Brighter
Energy (Low)	Fog Density	Denser
Spectral Centroid (Low)	Color Temperature	Warmer
Spectral Centroid (High)	Color Temperature	Cooler
Onset Strength (High)	Wind Strength	Stronger gusts
Scene Preset	Lighting + Fog Multipliers	Calm / Building / Intense / Transition

Notes

• unreal_bridge.py auto-spawns MusicDirectionalLight, MusicSkyLight, MusicFog, and MusicWind if they do not exist.
• To enable realtime sky capture, select MusicSkyLight and enable Real Time Capture in the Details panel.
• If Unreal logs "Could not find JSON", regenerate control_parameters.json by running steps 1-2 first.
• Avoid a trailing backslash when running the Unreal Python exec command.

File Structure

MTE/
  extract_energy.py       # Audio feature extraction
  map_parameters.py       # Feature-to-parameter mapping
  plot_energy.py          # Visualization utility
  unreal_bridge.py        # Unreal sequence generator
  generate_test_audio.py  # Synthetic test audio
  requirements.txt
  .gitignore

License

Research / Educational Use