Paper (arXiv) | Official Code Repository
Tianyi Hu · Qingxu Fu · Zhiqiang Pu · Yuan Wang · Tenghai Qiu
Note: This repository is a paper guide. The official source code lives at binary-husky/unreal-map. Head there for installation, issues, and contributions.
Unreal-MAP is an open-source Multi-Agent Reinforcement Learning (MARL) platform built on Unreal Engine. It lets researchers freely create custom multi-agent tasks using Unreal Engine's massive library of visual assets and physics tools, while providing a clean Python/Gym interface to plug in any MARL algorithm.
The key insight: existing MARL platforms either fix you to a set of built-in environments, or demand significant engineering to add new ones. Unreal-MAP removes this barrier by decoupling tasks, maps, and agents into independently recombineable modules.
| Challenge | Unreal-MAP's Answer |
|---|---|
| Building new environments is hard | Unreal Engine Blueprint scripting — no C++ needed for most tasks |
| Training is too slow | Controllable time-flow up to 32× acceleration; 1M+ physics frames/sec |
| Algorithms are tightly coupled to environments | Fully decoupled architecture; any algorithm plugs in via Gym API |
| Hard to scale to many agents | Tested up to 200+ agents with optimized perception pipelines |
| No real-world validation | Sim-to-real transfer demonstrated on physical UGVs and UAVs |
Unreal-MAP uses a five-layer hierarchical architecture:
Layer 5: Python Interface ← Gym-compatible API, algorithm integration
Layer 4: Advanced Modules ← Blueprint perception, appearance, kinematics
Layer 3: Base Classes ← Blueprint agents and entities
Layer 2: Specification ← C++ systems and standards
Layer 1: Native ← Unreal Engine assets and physics
On the Python side, the HMP (Hybrid Multi-agent Playground) framework handles parallel environment execution, multi-team information blocking, and JSONC-based configuration — so you only need to implement your algorithm logic.
The paper benchmarks 7 algorithms across 15 tasks in 4 scenarios:
Scenarios
| Scenario | Description |
|---|---|
| Metal Clash | Heterogeneous combat, ground + aerial units (5v5 up to 100 agents) |
| Monster Crisis | Cooperative defense against scripted waves (easy / medium / hard) |
| Flag Capture | Competitive multi-team flag objectives |
| Navigation Game | Adversarial pursuit-evasion |
Algorithms
- Actor-Critic: MAPPO, HAPPO, HATRPO
- Value-Based: QMIX, WQMIX, QPLEX, QTRAN
Highlights: Value-based methods excel in sparse-reward settings; MAPPO scales better at 50–100 agents; HAPPO/HATRPO lead in competitive multi-team tasks. 100K training episodes complete in ~3 hours.
Policies trained in simulation were deployed directly onto physical robots (UGVs + UAVs) using a motion capture system for feedback — with no additional fine-tuning. This validates Unreal-MAP as a practical tool for real-world robotics research.
Everything you need is in the official repository:
git clone https://github.com/binary-husky/unreal-map.git- Docker setup:
DOCS/setup_docker.md - Running experiments:
python main.py --cfg your_experiment.jsonc - Adding environments:
MISSION/env_router.py - Adding algorithms:
ALGORITHM/example_foundation.py
@article{hu2025unrealmap,
title={Unreal-MAP: Unreal-Engine-Based General Platform for Multi-Agent Reinforcement Learning},
author={Hu, Tianyi and Fu, Qingxu and Pu, Zhiqiang and Wang, Yuan and Qiu, Tenghai},
journal={arXiv preprint arXiv:2503.15947},
year={2025}
}