Disclaimer: This entire script was written by an LLM. It may be inefficient, error-prone, or simply not work for your use case. I'm sharing it because it worked well enough for my specific needs (Reverend Insanity, Ollama, qwen2.5:32b). Test it yourself, read the code, and don't trust it blindly. Contributions and bug reports welcome.

I run it with

python "C:\Reverend Insanity\summarize.py" --api-base "http://localhost:11434/" --chapter-dir "C:\Reverend Insanity" --output-dir "C:\Reverend Insanity\summaries" --end 735 --chunk-size 1 --timeout 600 --no-llm-compact --model "qwen2.5:32b"

Command Line Usage

usage: summarize.py [-h] [--api-base URL] [--api-key KEY] [--model MODEL]
                    [--chapter-dir DIR] [--output-dir DIR] [--chunk-size N]
                    [--num-ctx N] [--start N] [--end N] [--timeout SEC]
                    [--delay SEC] [--force] [--temperature FLOAT]
                    [--max-tokens N] [--archive-every N] [--archive-threshold N]
                    [--max-repair-retries N] [--context-events N]
                    [--max-traits N] [--max-knowledge N] [--max-goals N]
                    [--drop-dormant-threads] [--io-workers N]
                    [--migrate FILE] [--migrate-chapter N]
                    [--compact-every N] [--dry-run]
                    [--max-relevant-backfill N] [--ubiquity-threshold FLOAT]
                    [--max-relationships N]
                    [--python-compact] [--no-python-compact]
                    [--llm-compact] [--no-llm-compact]
                    [--category-compact-threshold N]
                    [--emergency-max-chars N] [--emergency-max-events N]
                    [--emergency-max-traits N] [--emergency-max-knowledge N]
                    [--emergency-max-goals N] [--emergency-max-threads N]
                    [--emergency-max-world N] [--emergency-max-gu N]

Quick Start

# Basic run — processes all chapters in current directory
python summarize.py --chapter-dir ./chapters --output-dir ./output

# Process a specific range
python summarize.py --chapter-dir ./chapters --output-dir ./output --start 100 --end 200

# Dry run — see what would be sent to the LLM without making any calls
python summarize.py --chapter-dir ./chapters --output-dir ./output --start 1 --end 1 --dry-run

# Resume from where you left off (skips already-summarized chapters)
python summarize.py --chapter-dir ./chapters --output-dir ./output

# Re-process everything from scratch
python summarize.py --chapter-dir ./chapters --output-dir ./output --force

# Use a different model
python summarize.py --chapter-dir ./chapters --output-dir ./output --model llama3:8b

# Use OpenAI-compatible API instead of Ollama
python summarize.py --chapter-dir ./chapters --output-dir ./output \
    --api-base https://api.openai.com/v1 --api-key sk-... --model gpt-4o

# Migrate an old v1 registry
python summarize.py --output-dir ./output --migrate ./old_output/registries/registry_ch0500.json --migrate-chapter 500

Key Options

Flag	Default	Description
--chapter-dir	.	Directory containing chapter XHTML/text files
--output-dir	./output	Where summaries, registry, and logs are saved
--model	qwen2.5:14b	LLM model name
--num-ctx	24576	Context window size (for Ollama)
--chunk-size	2	Chapters per LLM call
--start / --end	all	Chapter range to process
--dry-run	off	Preview what would be sent, no LLM calls
--force	off	Re-process even if summaries exist
--temperature	0.3	LLM sampling temperature
--max-tokens	2048	Max output tokens per call
--delay	2.0	Seconds between LLM calls
--compact-every	50	Registry compaction interval (0 disables)
--archive-threshold	250	Chapters before inactive entities are archived
--context-events	10	Recent events shown to the model
--max-relationships	40	Cap on relationships in context
--python-compact	on	Python-side pre-trimming before LLM compaction
--llm-compact	on	Per-category LLM compaction (Pass 2)

Output Structure

output/
├── registry/
│   ├── registry.json          # Master knowledge graph (all active entities)
│   ├── registry_archive.json  # Archived/inactive entities (full data preserved)
│   └── registry_backup.json   # Last registry before compaction
├── summaries/
│   ├── ch0001.md              # One summary per chapter (300-500 words)
│   ├── ch0002.md
│   └── ...
├── events/
│   └── event_log.json         # Full event log with participants and consequences
└── logs/
    └── processing.log         # Processing log with token estimates and errors

---