Download Ithaca.exe · double-click it · pick a model · chat
Chat with language models, show them pictures, hand them PDFs, and have them make images, video and speech. Let one work on the code and Office files in a folder, or put several on one hard question. Everything runs on your own PC, from one exe of about 3 MB.
There is no installer, no Python, no llama.cpp and no CUDA toolkit. The model loaders, tokenizers, transformer code, CPU and GPU kernels, the image, video and speech engines and the web UI are all written from scratch in C++. GPU code is generated as PTX at startup and compiled by the NVIDIA driver you already have.
- Download
Ithaca.exeand put it in its own folder. - Double-click it. A console window opens and the chat page opens in your browser.
- Click Models and download one, or point it at a folder of
.gguffiles. It also finds models from GPT4All and LM Studio on its own. - Load the model and start typing. You can type before it has finished loading.
Settings, chats and memories are kept next to the exe, so the folder is portable. Quit in the sidebar (or closing the console) stops everything and frees all memory.
Chat with a PDF. Long files are searched, and the answer points to pages.
Download models and check your PC. Each model shows whether it fits and roughly how fast it will run.
Open Coding in the sidebar, pick a folder, and ask for a change. The model works through a small set of tools, and you see each step as it happens.
- Real files. It lists, searches, reads, creates, edits, renames and deletes files in the folder you picked, and nowhere else.
- Office files too. Word, PowerPoint and Excel files are shown to the model as text with paragraph labels or cell addresses. Edits keep the formatting, and spreadsheet formulas recalculate when the file is opened.
- You stay in charge. Every change is shown as a diff to approve, or applied straight away if you turn on auto-apply. Commands (tests, builds) always ask first, have a time limit, and stop with Stop or Esc.
- Undo. Each change can be undone, or Undo all rolls back a whole task. Deleted files go to the Recycle Bin.
- Only this PC. The file tools answer requests from this computer only, never through
--hostor from another website. - Valid tool calls, even from small models. Once the model starts a tool call, Ithaca only lets it pick tokens that keep the JSON valid. With Gemma 3 4B, broken tool calls dropped from most calls to about 1 in 27.
- Doesn't get stuck. When the model repeats the same step with the same result, Ithaca tells it to try something else, and stops the task if it keeps going.
- Chats are kept per folder. Each folder's chat is saved in full next to the exe, so it comes back when you switch folders. Delete one from the sidebar; the folder itself is never touched.
- Long tasks fit. When a task nears the context limit, older steps are compacted into a summary and the newest steps stay word for word. This works in Chat too, and both can be turned off under Settings → General.
Models trained for tool calls work best. Qwen3-Coder 30B-A3B runs at about 15–18 tokens a second on a 12 GB card, with the rest of the model in RAM.
The test bench. eval/agent_eval.py gives the agent 8 small tasks (fix a bug, add a feature, rename a function, edit a Word file and a spreadsheet, change a config, write a README, answer a question). Each task has a hidden check, so pass rates are measured, not guessed. You can also compare system prompts. eval/chat_eval.py does the same for the Chat page with 12 auto-checked questions. Running these benches is what pushed the agent from 38% to 75%: most of the gains came from fixing how Ithaca reads tool calls, not from changing the prompt.
Open Orchestration, tick the models you want, pick a judge, and ask. Each model works through the request on its own: it writes several answers, reviews them, and refines them over several rounds, keeping, editing, deleting or adding answers until they hold up. Then the judge compares each model's best answer.
The steps change with the kind of request. Auto reads the request and picks one of these:
| Method | Each model | The judge |
|---|---|---|
| Hard question | 3 answers, criticized and refined for at least 4 rounds (up to 6) | merges the strongest points |
| Code | 3 solutions; Python is run, and the output goes into the reviews; 2–4 rounds | picks the solution that works |
| Math & logic | 3 independent solutions, every step re-checked; 2–4 rounds | holds a vote on the final answers, then writes up the winner |
| Creative writing | 3 drafts in different styles, edited like an editor would; 3–4 rounds | picks the strongest piece |
| Ideas & plans | 3 plans; weak ones dropped, overlaps merged, new ones added; 3–4 rounds | combines them into one plan |
| Quick facts | one answer | writes one answer and notes where the models disagree |
You see everything: every answer with its score and whether it was kept, edited, deleted or added, every round of review, and which model the best answer came from. Runs are saved next to the exe and listed in the sidebar. Models run one after another (the judge goes last, so it's still loaded), so a run with two small models takes about 10 minutes.
Ithaca generates at 96–104% of llama.cpp's speed and reads prompts at 85–116% of it, on the same files and the same GPU.
On mixture-of-experts, latent-attention and Mamba-hybrid models Ithaca generates faster than llama.cpp (101–125%); prompts are at 75–87% of it.
Low-bit files fit bigger models in less VRAM. Ithaca runs all of them on the GPU at their compact size.
Chat (GGUF): Llama 2/3, Mistral, Mixtral, Qwen 2/2.5/3, Qwen2-MoE (Qwen1.5-MoE), Qwen3-MoE, Qwen3.5, Phi-3/4, Granite, Granite-MoE, Granite 4.0 H, Gemma 2, Gemma 3, gpt-oss, OLMo 2, OLMoE, SmolLM3, LFM2, ERNIE 4.5, DeepSeek V2 and V2-Lite and the DeepSeek-R1 distills. Split GGUFs load from their first part.
Quantizations: F32, F16, BF16, Q4_0, Q4_1, Q5_0, Q5_1, Q8_0, Q2_K to Q6_K, IQ1_S to IQ4_XS, and MXFP4 (gpt-oss). Low-bit weights stay that small on the GPU.
Pictures in: put the model's mmproj file next to it (for example gemma-3-4b-it-Q4_K_M.gguf with mmproj-model-f16.gguf) and Ithaca picks it up.
| Models | Notes | |
|---|---|---|
| Images | SD 1.5, SD 2.x, SDXL single-file checkpoints and their fine-tunes; Flux.1 schnell and dev (all-in-one fp8 files) | DPM++ 2M, Euler, Euler a; optional separate VAE. Flux on a 12 GB card: about 1.5 min at 512x512, 3 to 5 min at 1024x1024 |
| Video | Wan 2.2 TI2V-5B, Wan 2.1 T2V (1.3B), Wan 2.2 T2V A14B (GGUF high/low-noise pair) | Needs the umT5 encoder and the matching Wan VAE; saves MP4. A14B on 12 GB: about 480x272, 6 to 10 min for 2 s |
| Speech | F5-TTS + Vocos | Voice cloning from a 5 to 12 second clip |
Not supported yet: DeepSeek V3/R1 (full size), Llama 4, SD3, LoRAs, image-to-video. Settings → Model shows whether each of your files is supported, and Settings → Download models gets image, video and speech models with everything they need.
Why use it instead of llama.cpp, Ollama or LM Studio?
- One 2 MB file: no install, no service, no runtime downloads.
- Images, video and speech out, and pictures and documents in, all in the same chat.
- Models bigger than your VRAM still run, and you can type before loading finishes.
- Speculative decoding is automatic: with a small model of the same family installed (Qwen3 0.6B next to Qwen3 14B), it drafts tokens for the big one. Qwen3 14B writes code at about 44 tok/s this way.
- If you need Linux, Mac or AMD support, llama.cpp is the better choice today.
How do models bigger than my GPU run?
- Opening a model only reads its header, so you can type right away while layers move up in the background.
- Weights reach the GPU through pinned buffers and are converted there by a kernel.
- Follow-up messages reuse the cache for the part of the chat that didn't change, and the cache is 8-bit so long chats stay small.
- When an image or video job needs the GPU, the chat model steps aside and comes back afterwards.
Which quant should I download?
Start with Q4_K_M. If it doesn't fit, IQ4_XS is smaller at the same speed, and the IQ3 and IQ2 files let much bigger models fit on small GPUs at some cost in quality. Every speed is in docs/quant-speeds.md.
What hardware do I need?
Windows 10 or 11. The NVIDIA driver you already have is enough: no CUDA toolkit. A 4B model at 4-bit needs about 2.5 GB of VRAM; a 14B model about 8 GB.
How do I use it from my own code?
Ithaca.exe --server --model file.gguf starts an OpenAI-compatible API on port 8080 with no window.
| Endpoint | What it does |
|---|---|
GET /v1/models |
The loaded model and every GGUF Ithaca can find |
POST /v1/chat/completions |
Chat, streaming or not. Images as image_url data URLs (vision models), stop strings, and "model" switches to another local model. Reasoning models return their thinking in reasoning_content. |
POST /v1/completions |
A single prompt |
POST /v1/images/generations |
Images from text: size, n, steps, guidance_scale, negative_prompt, seed, response_format |
POST /v1/images/edits |
Edit a picture: image (base64 or multipart), optional mask (white = change), strength |
POST /v1/audio/speech |
Text to speech in one of your voices (returns WAV) |
POST /v1/videos/generations |
Text to video (Wan 2.2): prompt, seconds, size |
GET /health |
Status |
curl http://127.0.0.1:8080/v1/chat/completions -d '{"messages":[{"role":"user","content":"Hello!"}],"stream":true}'from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="none")
print(client.chat.completions.create(model="default", messages=[{"role": "user", "content": "Hello!"}]).choices[0].message.content)Options: --port 8080, --host 0.0.0.0 to serve other devices on your network, and --api-key KEY (or the ITHACA_API_KEY variable) to require Authorization: Bearer KEY. In server mode only the API is reachable; the app's own endpoints are switched off.
/v1/ithaca/park and /v1/ithaca/wake move the chat model to RAM and back in about half a second, to free VRAM for something else. In server mode only the API is reachable.
How was the speed measured?
RTX 3060 12 GB, 16 GB RAM, the same GGUF files, all layers on the GPU, 3 runs each, the two programs run back to back per model. llama.cpp is build 11146 with CUDA, run as llama-bench -p 512 -n 128 -ngl 99 -r 3; Ithaca as Ithaca.exe --bench --model file.gguf, which runs the same test. Raw numbers: docs/bench-results.json.
Prompts are read with int8 tensor cores (the same idea as llama.cpp's "mmq"), and the norm, residual add and int8 conversion around each matmul run as one fused step.
Command line options
Ithaca.exe [--model file.gguf] [--ctx N] [--vram MB|0] [--ram MB] [--threads N] [--port 7860] [--no-browser]
Ithaca.exe --model file.gguf --prompt "Hello" [--image photo.jpg] [-n 128]
Ithaca.exe --bench --model file.gguf
Ithaca.exe --extract report.pdf
Ithaca.exe --server [--model file.gguf] [--host 0.0.0.0] [--port 8080] [--api-key KEY]
Building from source
You need MinGW-w64 g++. Run build.bat; the result is a single static Ithaca.exe. The CPU path needs AVX2.
src/model.* transformer, MoE, tier planner, CPU and GPU forward passes
src/cuda_tier.* NVIDIA driver wrapper, quantized matmul kernels, weight loader
src/gx*.cpp/h GPU runtime for the media engines and prompt processing
src/sd.*, wan.*, tts.* image, video and speech engines
src/vision.* image input (SigLIP encoder for Gemma 3)
src/docs.cpp PDF / DOCX / PPTX / XLSX text extraction
src/chat.* chat templates, sampling, prompt cache
src/hub.cpp model downloads, system info, benchmark
src/main.cpp HTTP server and CLI; src/ui.cpp the web UI
- Close the rest of the prompt-speed gap on Gemma 3 and mixture-of-experts models.
- Faster gpt-oss prompts; DeepSeek V3 routing.
- SD3, LoRAs, image-to-video; faster Flux and Wan 14B on 12 GB cards.
- AMD and Intel GPUs (Vulkan).
- Orchestration: run small models at the same time when they fit in VRAM together.
Apache 2.0. See LICENSE.





