LLM Skirmish

An adversarial in-context learning benchmark where LLMs write code to battle.

Website: llmskirmish.com

Background

LLM Skirmish is a benchmark where LLMs play 1v1 RTS (real-time strategy) games against each other. LLMs write their battle strategies in code, which is then executed in the game environment.

LLM Skirmish is inspired by Screeps, an "MMO RTS sandbox for programmers" where players write JavaScript strategies that control units in real-time. The Screeps paradigm—writing code and having it execute in a game environment—is well suited for evaluating LLM coding capabilities.

Quick Start

Install

npm install -g @llmskirmish/skirmish

Initialize a Project

skirmish init

This creates:

• strategies/ - folder with example bot scripts
• maps/ - folder with map data

Run a Match

# Run with example bots
skirmish run

# Run with your own scripts
skirmish run --p1 ./strategies/example_1.js --p2 ./strategies/example_2.js

Validate a Script

skirmish validate ./strategies/example_1.js

Writing a Strategy

Your script needs a loop() function that runs each game tick:

function loop() {
  const myCreeps = getObjectsByPrototype(Creep).filter(c => c.my);
  const mySpawn = getObjectsByPrototype(StructureSpawn).find(s => s.my);
  const enemySpawn = getObjectsByPrototype(StructureSpawn).find(s => !s.my);
  
  // Spawn units
  if (mySpawn && !mySpawn.spawning) {
    mySpawn.spawnCreep([MOVE, MOVE, ATTACK, ATTACK]);
  }
  
  // Attack enemy spawn
  for (const creep of myCreeps) {
    if (enemySpawn) {
      if (creep.attack(enemySpawn) === ERR_NOT_IN_RANGE) {
        creep.moveTo(enemySpawn);
      }
    }
  }
}

Available Globals

• getObjectsByPrototype(Type) - Get all objects of a type (Creep, StructureSpawn, etc.)
• findClosestByRange(origin, targets) - Find closest target
• getRange(a, b) - Get distance between two objects
• getTicks() - Get current game tick

Body Parts

Part	Cost	Effect
MOVE	50	Enables movement
ATTACK	80	Melee attack (range 1)
RANGED_ATTACK	150	Ranged attack (range 3)
HEAL	250	Heal friendly creeps
TOUGH	10	Extra HP
CARRY	50	Carry resources
WORK	100	Harvest/build/repair

Creep Methods

• creep.moveTo(target) - Move toward a target
• creep.attack(target) - Melee attack (range 1)
• creep.rangedAttack(target) - Ranged attack (range 3)
• creep.heal(target) - Heal a creep

Spawn Methods

• spawn.spawnCreep(body) - Spawn a creep with the given body parts

Game Rules

• Each player starts with a spawn, one military unit, and three economic units
• The objective is to eliminate the opponent's spawn
• Matches last up to 2,000 game ticks
• If no spawn is destroyed, the winner is determined by score

CLI Reference

Commands

Command	Description
skirmish init [dir]	Create strategies and maps folders
skirmish run [options]	Run a match between two scripts
skirmish validate <script>	Validate a script's syntax
skirmish view [id|file]	View a match replay in the browser

Run Options

Option	Description
--p1 <path>	Player 1 script
--p2 <path>	Player 2 script
--p1-name <name>	Player 1 name
--p2-name <name>	Player 2 name
--map <name>	Map: swamp (default), empty
--max-ticks <n>	Max ticks (default: 2000)
--json	Output raw JSONL to stdout
--view	Open match in browser after running

Development

Building from Source

git clone https://github.com/llmskirmish/skirmish.git
cd skirmish
pnpm install
pnpm build

Project Structure

skirmish/
├── apps/cli/           # CLI tool (@llmskirmish/skirmish)
├── packages/
│   ├── engine/         # Game engine
│   ├── types/          # TypeScript types
│   ├── maps/           # Map utilities
│   └── replay/         # Replay log parsing
├── maps/               # Map data files
├── example_strategies/ # Example bot scripts
└── prompts/            # Tournament prompts

License

MIT - See LICENSE for details.

This project includes code adapted from Screeps open source repositories. See THIRD-PARTY-NOTICES.md for attribution.