TestCrew

AI-powered UI testing. Point TestCrew at a website, describe test cases in plain English, and an AI agent drives a real headless browser to verify them — reporting pass/fail with screenshots, a live step-by-step trace, and its reasoning.

"Clicking the Blogs link in the navbar redirects to the blogs page." → ✅ passed in 25s, with evidence.

Requirements

• Node.js ≥ 22, pnpm
• Docker (for Postgres)
• The Claude Code CLI installed and logged in (claude). TestCrew uses it as the AI provider via ai-sdk-provider-claude-code, so there is no API key — model access runs on your Claude subscription.

Setup

pnpm install
pnpm exec playwright install chromium   # headless browser for the agent
docker compose up -d                    # Postgres 17 on localhost:5433
pnpm db:migrate                         # create the schema
pnpm dev                                # → http://localhost:3001

Using it

1. Create a project — a name and the site's URL.
2. Add test cases in plain English under Tests, or hit ✨ Suggest tests to have the AI crawl the site and propose some (you review before anything is added).
3. Run all (or ▶ Run a single test) and watch the live run report: every agent action, its commentary, and screenshots stream in as they happen. Cancel anytime.
4. The same page is the permanent report — verdicts always cite concrete observed evidence (URLs, titles, visible text). Open any test to step through what it did, with each step seeking the session recording to that moment.
5. Overview tracks pass rate, runs, failures and duration over time; Results lists every run; Memory shows the navigation the agent has learned about your site and lets you forget any of it.

Testing behind a login

Set TESTCREW_SECRET_KEY in .env (openssl rand -base64 32), then open Settings → Authentication → Sign in to capture. A real browser window opens on your machine; you log in by hand and close it.

TestCrew never asks for or stores your password. You do the signing in, so "Continue with Google", GitHub, Okta, SSO, magic links, TOTP, hardware keys and passkeys all work — none of them need provider-specific support, and none of them trip the bot detection that blocks automated sign-in. What's kept is the resulting session (cookies, localStorage, IndexedDB, passkeys), encrypted with your key, never sent to the model and never served over /api/media. Every test then opens already authenticated, spending zero agent turns on logging in.

Sessions expire. TestCrew checks yours once before a run starts and stops with "the captured session is no longer valid" rather than reporting a screenful of failures that are really one expired cookie. Re-authenticating reuses a saved browser profile, so the identity provider usually recognises the device and skips a second full 2FA challenge.

Anything typed into a password, OTP or card field is scrubbed from the ARIA snapshots sent to the model and from the step trace written to Postgres — Playwright reports input values verbatim, so this is enforced rather than assumed (pnpm exec tsx scripts/verify-auth.ts).

How it works

• Engine (src/engine/): each test gets a fresh headless Chromium context (1280×720). The Claude Code CLI runs the agent loop; the browser tools (navigate, click, fill, snapshot, screenshot, …) are bridged to it as an in-process MCP server. Page state is fed as ARIA snapshots with [ref=eN] element references. Guardrails: domain allowlist, 24-turn cap, 180s timeout, strict tool whitelist, evidence-cited verdicts.
• Runner (src/server/runner.ts): one run at a time, tests execute sequentially; every step is persisted to Postgres first and then pushed to the UI over SSE, so refreshing mid-run loses nothing.
• App: TanStack Start + React + Tailwind, Drizzle + Postgres, Vercel AI SDK v7. The UI is an app shell (sidebar + breadcrumbs) over per-project sections; charts are hand-rolled SVG driven by the same design tokens as everything else, so there is no charting dependency.

Dev commands

pnpm spike <url> "<test case>"        # run the engine standalone, no UI
pnpm exec tsx scripts/verify-run.ts   # end-to-end pipeline check (real run, persisted)
pnpm exec tsx scripts/verify-auth.ts  # session sealing, secret redaction, session replay
pnpm typecheck
pnpm db:studio                        # browse the DB

License

MIT — see LICENSE.