ctk prints how many tokens a file weighs in an LLM context window. Like wc, but it counts model tokens, not words. The output is a bare number on stdout, so it pipes into anything — shell guards, git hooks, agent scripts.

Install

One command with Bun 1.0 or newer:

bun install -g github:psychosomat/ctk

Usage

ctk file.txt        # count a file
cat file.txt | ctk  # or count stdin

Uninstall / develop on the source

bun remove -g ctk   # remove a global install

git clone https://github.com/psychosomat/ctk && cd ctk
bun install && bun link   # run your local clone as `ctk`

Pipe it anywhere

stdout carries one integer per input file. That is the entire interface:

ctk prompt.md                # cost of one file
git diff | ctk               # cost of a patch
find . -name '*.md' -exec ctk {} + | awk '{s+=$1} END {print s}'   # a whole folder

That makes it a gauge for anything that spends a context window:

# exit 0 only if the file fits a 4k-token budget
ctk notes.md | awk '{exit ($1 <= 4096) ? 0 : 1}'

# fail a hook when a prompt file outgrows its budget
[ "$(ctk system-prompt.md)" -le 8000 ]

Behavior contract

• Input must be UTF-8. Invalid byte sequences decode to U+FFFD before counting.
• With no argument and a terminal attached, ctk prints usage: ctk <file> to stderr and exits 1.
• A missing file prints the error to stderr and exits 1.
• Multiple files via find -exec ctk {} + print one number per line, in argument order.

Why it holds up

Exact, not estimated Real BPE encoding with the o200k_base ranks — the same tokenizer OpenAI ships in tiktoken. Counts match the official Python implementation token for token.	Unix-grade A number on stdout, errors on stderr, exit codes you can script against. No banners, no progress bars, no config.
Fully offline The BPE ranks ship inside the js-tiktoken dependency. Zero network calls at runtime, startup to result in about half a second.	Auditable in one screen The entire program is a 22-line Bun script. There is nothing else to trust — read it and you know what it does.

That's the entire program

#!/usr/bin/env bun
import { readFileSync } from "node:fs";
import { Tiktoken } from "js-tiktoken/lite";
import o200k_base from "js-tiktoken/ranks/o200k_base";

function readInput(): string | null {
	const path = process.argv[2];
	if (path) return readFileSync(path, "utf8");
	return process.stdin.isTTY ? null : readFileSync(0, "utf8");
}

try {
	const text = readInput();
	if (text === null) {
		console.error("usage: ctk <file>");
		process.exit(1);
	}
	console.log(new Tiktoken(o200k_base).encode(text).length);
} catch (error) {
	console.error(`ctk: ${error instanceof Error ? error.message : error}`);
	process.exit(1);
}

What it counts

ctk uses the o200k_base byte-pair encoding from OpenAI's tiktoken.

Encoding	Applies to	Counts match?
o200k_base (this tool)	GPT-4o, GPT-4.1, o-series	yes — what ctk reports
cl100k_base	GPT-4, GPT-3.5-turbo	close, but not identical
other tokenizers	Claude, Gemini	differ

Verify parity with the official Python tiktoken

Encode the same file with both tools and compare the numbers:

python3 -m venv .venv
.venv/bin/pip install tiktoken
.venv/bin/python -c "import tiktoken; print(len(tiktoken.get_encoding('o200k_base').encode(open('file.txt', 'rb').read().decode('utf-8'))))"
ctk file.txt

Encoding is also lossless: every input survives encode and decode unchanged.

Development

bun run lint        # biome check
bun run typecheck   # tsc --noEmit

License

MIT