Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 63 additions & 1 deletion .claude/skills/ringer/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,8 @@ description: >-
are about to do a "quick check" that spawns a model or a CLI agent; you are
reviewing or diagnosing failed worker or model output; you catch yourself
thinking a task is "small enough to just do myself" — that thought IS the
trigger (a single task is a one-task manifest); or you are writing or
trigger (a single task is a one-task manifest, and a bounded read-only
question is `ringer.py ask`); or you are writing or
reviewing a manifest, choosing a swarm pattern (review swarm, fix swarm,
focus group, bakeoff, research-with-proof), picking a worker engine, or
debugging a failed run. SKIP only for: reading or searching files, git
Expand Down Expand Up @@ -68,6 +69,41 @@ Lint catches unverifiable checks, silent checks, worktree deliverable/commit
loss, serial fan-out, write collisions, and underspecified specs; `run`
prints the same findings as non-blocking warnings.

## The one exception: `ask`

Rule 2 holds for anything that changes a file, runs a build, or produces an
artifact worth checking. One lane doesn't fit it: the human asks a bounded,
read-only question over source you can already point at, and the answer is
prose. A manifest for that is ceremony — but answering it in your own context
means pulling whole files into a conversation that is already expensive.

```bash
./ringer.py ask "<the human's request>" --source /absolute/path/to/source
```

`ask` selects the passages that match the request, caps the packet, spawns one
clean worker on it, and allows a single attempt. Repeat `--source` for several
files or directories; `--state` takes a small file of settled decisions;
`--dry-run` shows you the packet and spends nothing. If everything that matched
is too large for the packet it says so and stops before the model call rather
than letting a worker guess — but a source small enough to fit whole is sent
whole, relevant or not, so choosing the sources IS the work. Directory scans
stay inside the tree you name; a symlink leading out of it is skipped and
reported. Runs appear on Ringside like any other, and `--redact` hides the
request from Ringer's own state and eval records — it cannot scrub raw worker
output, which is captured verbatim by design.

**Be honest about what it verifies.** The check is that `answer.md` exists and
is non-empty. That is the weakest check in the tool, and it is also the best
available — there is nothing to execute against free-form prose. `ask` proves
the worker answered, never that the answer is right. You still read it.

**Everything else is a manifest.** Code changes, external actions, research
you intend to act on, anything whose output a check could actually execute —
those keep the full path. When a request sits near the line, the tiebreaker is
whether you could write a check that would catch a wrong answer. If you can,
write it, and make it a manifest.

## One job, one artifact

A job the human asked for — however many rounds it takes — is ONE artifact.
Expand Down Expand Up @@ -327,6 +363,32 @@ someone's untracked scratch files.
numbers took care of themselves — every attempt already landed in the
local model log (`./ringer.py models` to see the updated scoreboard).

## Spend your own context deliberately

The scoreboard exists so that worker tokens buy evidence. Your own tokens are
not free either, and nothing in the tool constrains them:

- **Reach for code before a model.** Counting, sorting, exact-text search,
field extraction, format conversion, file comparison, validation — `rg`,
`jq`, a parser, a two-line script. A model imitating `grep` is an expensive
way to get a worse `grep`.
- **Select passages; don't load files.** Search first, then read what matched.
Loading a whole transcript because the answer is somewhere inside it is how
a cheap question turns expensive. `ask` does this for you; when you are not
using `ask`, do it by hand.
- **Load a tool when the job needs it** — not every connector and schema at
the top of a session on the chance that one gets used.
- **Answer the question that was asked.** A sentence when a sentence was asked
for. No process diary, no restating the human's request back to them, no
unrequested options.
- **Never retry into a limit.** A token- or usage-limit failure is not a
transient error; retrying it just burns the budget faster. Reduce the input
or take a cheaper path.

When you claim a saving, count the whole job — every call, including your own
planning and review. Moving tokens from your context into a worker's is only a
saving if the total came down.

## Baked-in invariants (preserve in any change to ringer.py)

Stdin closed (`< /dev/null`); sandbox mode explicit; verification executes
Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Honestly: we never asked for contributions, and the number of people showing up
1. **Small and scoped — one feature or fix per PR.** The single biggest predictor. Four PRs merged same-day the week this guide was written; the two large bundles (52 files; 16 files) were both sent back for splitting regardless of quality. If your change has an "and," consider splitting it. Resist scope creep in your own diff: drive-by refactors, stale copies of main, and bonus features all slow the part we want.
2. **Rebased on current main.** Main moves fast here. A conflicting PR can't be audited.
3. **Executed proof for every claim.** A test that runs beats a screenshot; a check that prints *why* it fails beats a silent `exit 1`. CI runs the full suite on macOS and Linux (required) and a non-blocking `windows-latest` harness — platform claims must be proven by the job for that platform, not asserted.
4. **Match the house style.** Single-file `ringer.py`, stdlib only, Python 3.11+, frozen dataclasses, tests in `tests/` runnable by `python3 -m unittest discover -s tests`. Set `RINGER_NO_SELF_UPDATE=1` in tests that spawn the CLI.
4. **Match the house style.** Single-file `ringer.py`, stdlib only, Python 3.12+, frozen dataclasses, tests in `tests/` runnable by `python3 -m unittest discover -s tests`. Set `RINGER_NO_SELF_UPDATE=1` in tests that spawn the CLI.
5. **Real motivation.** PRs that fix an observed failure (say so in the description — "burned 100k tokens against a broken check" is a great opening line) review better than speculative hardening.

## Ringside UI contributions — actively encouraged
Expand Down
40 changes: 38 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ manifest.json ──▶ ringer.py ──▶ N parallel workers (codex exec, each

## Quickstart

Ringer runs on macOS and Linux (Windows via WSL) and needs Python 3.11+.
Ringer runs on macOS and Linux (Windows via WSL) and needs Python 3.12+.

1. Install a worker CLI and sign in (Codex is the built-in default engine):

Expand Down Expand Up @@ -102,6 +102,8 @@ Each task gets its own directory, its own worker, its own log, and its own verdi
| `model` | Which model a harness engine runs for this task — fills the engine's `{model}` placeholder (e.g. `"openrouter/moonshotai/kimi-k2.7"`); empty uses the engine's `model_default` |
| `task_type` | Optional free-form string naming the kind of work this task is, so the model-performance log can slice pass rates by task shape rather than only by model. Suggested vocabulary: `code-feature`, `code-fix`, `code-review`, `test-hardening`, `docs`, `research`, `persona-review`, `copywriting`, `site-build`, `motion-design`, `image-gen`, `data-pipeline`, `format-conversion`, `probe`, `bakeoff`. Empty is allowed; the log just reports it under `(none)`. |
| `timeout_s` | Per-task kill timer (default 900) |
| `max_attempts` | How many times this task may run (default 2 — one try plus one retry with the check's failure output injected). Set `1` for a hard no-retry lane |
| `redact_spec` | Replace this task's spec with `[redacted request packet]` in the run state, the logged command line, and the eval row, for specs carrying sensitive material. Redacts Ringer's own records only — captured worker output is never rewritten (invariant), so a worker that echoes its request still puts that text in `worker.log` |
| `engine_args` | Extra CLI flags for this task's worker, spliced in at the engine's `{engine_args}` placeholder — e.g. `["-c", "model_reasoning_effort=low"]` so the orchestrator picks reasoning depth per task |
| `verified` | One plain-English sentence saying what the check proves — shown on the results page next to "finished & checked" |
| `full_access` | Worker runs unsandboxed — required for workers that spawn their own sub-workers; must also be enabled in config |
Expand All @@ -111,6 +113,39 @@ Each task gets its own directory, its own worker, its own log, and its own verdi

Not sure what your tasks even are yet? [`docs/interview-prompt.md`](docs/interview-prompt.md) is a prompt you paste into any chatbot; it interviews you about the job and hands back a brief your orchestrating agent can turn into a manifest. Ready-made skeletons for the patterns that work live in [`templates/`](templates/).

## `ask` — one bounded question, one clean worker

Not every question deserves a manifest. When you want a read-only answer over
source you can already point at, `ask` selects the passages that match the
request, caps the packet, and runs a single worker on it:

```bash
./ringer.py ask "Why did the Wednesday release slip?" --source notes/status.md
./ringer.py ask "..." --source src/ --source docs/ --dry-run # show the packet, spend nothing
```

Repeat `--source` for more files or directories. `--state` takes a small file
of settled decisions and is preferred over ordinary sources when the packet is
tight. `--max-packet-bytes` sets the budget (default 16,000). `--dry-run`
prints the selection report and stops before any model call. `--redact` keeps
the request out of the run state and eval row. The run appears on Ringside and
in the artifact library like any other.

If everything that matches is too big for the packet, `ask` says so — naming the
budget you'd need — and stops **before** calling a model. It never sends an
empty packet. A source small enough to fit whole is included whole, whether or
not it looks relevant, so pointing `ask` at unrelated material still costs one
call: the packet is only as good as the sources you name.

Directory scans stay inside the tree you named. A symlink pointing out of it, or
one resolving to a sensitive filename, is skipped and reported. A file you name
explicitly is always read — naming it is consent.

> `ask` verifies only that an answer was produced and is non-empty. There is
> nothing to execute against free-form prose, so this is the one lane in Ringer
> where the check does not prove the result is right. Read the answer. Anything
> whose output a check could actually execute belongs in a manifest.

## Lint

Lint checks a manifest for the mistakes that make swarms hard to trust: checks that cannot fail, silent checks, worktree deliverables that disappear, worker commits that die with deleted worktrees, serial fan-out, write collisions, and underspecified specs.
Expand Down Expand Up @@ -366,7 +401,8 @@ Contributions are welcome — see [CONTRIBUTING.md](CONTRIBUTING.md) for the phi

## Requirements

- Python 3.11+ (stdlib only; `psycopg` needed only for the optional Postgres eval backend)
- Python 3.12+ (stdlib only; `psycopg` needed only for the optional Postgres eval backend)
- **Changed:** the supported floor moved from 3.11 to 3.12. CI has only ever run 3.12, so 3.11 was a promise nothing enforced — the honest fix is to state the version we actually test. Today's code still happens to run on 3.11; that is no longer guaranteed, and 3.11 breakage won't be treated as a bug.
- At least one agent CLI (Codex works out of the box)
- Rust toolchain, only if you're building Ringside from source

Expand Down
7 changes: 7 additions & 0 deletions docs/MODEL-NOTES.md
Original file line number Diff line number Diff line change
Expand Up @@ -549,3 +549,10 @@ the fix and used the engine_args splice; later runs should use the `model` field
honest work ("corrected ON 2026-07-17", a prose cross-reference counted as a
duplicated section) — strict-on-substance/tolerant-on-format cuts both ways.


## opencode (harness note, any model)
- 2026-07-28 (code-review, pr82-token-saver-review): GLM 5.2 produced a complete, high-quality 218-line report but could NOT write it to an output directory created by the parent Claude Code process — every write returned EPERM. It then spent ~3000s burning retries on ctypes/`openat`/AppleScript/`sandbox-exec` workarounds until it timed out, and the task logged as FAIL despite the deliverable existing in its taskdir. Codex workers in the same run were unaffected. Lesson: point opencode workers' output INSIDE their own taskdir and harvest via `expect_files`; never hand them a shared output dir another process created. This is an orchestrator spec bug, not a model failure — do not read the FAIL as evidence against GLM.

## Process lessons (2026-07-28, PR #82 review)
- **Ideas worth keeping from a rejected PR.** PR #82's pre-call gateway was dropped (needs your own API key, so it converts flat-rate OAuth plans into metered API billing; incompatible with Claude Code; and it saves tokens by stripping the tool list, which is the thing that makes the CLI worth using). One idea inside it is worth remembering if the problem ever comes back: an *explicitly blessed* answer cache — key a reviewed answer to the exact request plus the exact selected source packet, and replay it with zero upstream calls, never auto-accepting a model answer. It only fires on byte-identical repeats, which is why it didn't justify 2,000 lines here.
- **Doc-stated support floors need a CI job or they are fiction.** README promised Python 3.11+ while CI only ever ran 3.12; a 3.12-only f-string reached review with a fully green suite. Either test the floor or move it.
Loading
Loading