I run large models on hardware that shouldn't really manage it, and I write down what actually happens.

That combination is why I can run a 758B model off disk. It is also why my numbers tend to disagree with everyone else's.

Building a local AI ecosystem: several agents sharing one Obsidian knowledge base, a full-duplex voice model, and a 758B MoE running on a single consumer GPU. Everything measured, nothing assumed. Most of it lives on my own machine rather than here.

I care about one thing more than speed: a model that says "I have not checked" instead of a confident wrong number.

JustVugg/colibri - pure-C MoE inference

• #1042 - datapoint.py hardcoded RAM to 8 GB on Windows. It sizes the cache eviction, so every Windows "cold" benchmark ever published by that tool was measured warm. Fixed and merged.
• #837 - Windows + CUDA build, and helping other people get it running.

The RTX 3090 is still missing from their benchmark table. Working on it.

Open an issue anywhere I'm active, or start a discussion. I answer.