Deterministic front-end validation with evidence an auditor can sign.
In regulated software (medical devices, pharma, other GxP systems), display checks are often still done by hand: follow a test script, take a screenshot, paste it into a document, compare it with the expected value, sign. validshot does the clicking, the screenshots and the comparisons. People still write, approve and sign.
57-second demo on YouTube (with sound): a test app with planted defects, the execution record that catches them, the run on video, and a signed, timestamped summary report.
Step 2 of a sample execution record, run against a test app with planted defects. The stock badge's DOM text is the expected "Low stock", but a banner covers it, so the check fails: a tester looking at the screen couldn't read it either. The product name is cut off, and the total is $569.90 where 3 × $189.99 = $569.97.
A protocol (YAML) lists steps: what a tester does, the requirement it covers, and what must be displayed. validshot executes it in a pinned browser (Chromium by default; Firefox or WebKit if the protocol says so) and produces three kinds of document. All of them are in docs/sample/, generated from the example protocol and unedited.
1. A review copy for the approver (PDF).
The protocol laid out like a manual test script, for reviewers who don't read YAML.
Approval (validshot lock) pins the protocol's exact bytes.
2. An execution record for every run (FAIL, PASS). For every step: the raw screenshot (never modified), an annotated copy with each check boxed pass/fail, and expected vs actual for every check, as in the image above. The record (PDF + JSON) also holds requirement traceability, deviations and signature blocks. Every file is SHA-256 hashed into an append-only, hash-chained audit log.
3. A summary report over all formal runs (PDF). Results by protocol, traceability, the full execution history including failed runs, every deviation, and open items. Each run's evidence is re-verified when the report is generated.
Built for regulated software (IEC 62304 / SaMD, GxP CSV/CSA, GAMP 5). Design rationale: docs/design.md.
- No AI in the execution path. An agent may help write a protocol (MCP tools below). A formal run is a deterministic replay. It calls no LLM, does no selector healing and no automatic retries. A target that is not found is a FAIL.
- People approve, people execute. A reviewer locks a protocol (
validshot lock), which pins its exact bytes. A formal run refuses an unlocked or edited protocol and requires a named initiator. The MCP server has no lock tool and no formal-run tool. - Evidence is never touched. Raw screenshots are stored as captured. Annotations are drawn on a copy, in a separate browser context, so the app under test is never modified for the picture.
- "Displayed correctly" is more than the DOM. Every text or visibility assertion
also checks that the element is in the viewport, not covered by another element,
not truncated, and legible in the rendered pixels. A badge hidden under a banner
FAILs even if its DOM text is right. So does text under an overlay that hit-testing
cannot see (
pointer-events: none), or text in the colour of its background. - Failures are recorded, not hidden. A failed assertion is recorded as FAIL with its actual value, and the run continues. A failed action FAILs its step and marks every later step BLOCKED.
npm install -g validshot # or per project: npm install --save-dev validshot
npx playwright install chromium # once, if not already cached
validshot --helpPin the exact version in a validated setup (npm install validshot@0.3.0): the tool
version goes on every record, and a version change needs its own assessment.
git clone https://github.com/boheling/validshot && cd validshot
npm install
npx playwright install chromium # once, if not already cached
npm run ka-app # known-answer test app on :4173 (/good and /bad)
validshot check examples/ka-app/protocols/KA-001-order-form.yaml
validshot run examples/ka-app/protocols/KA-001-order-form.yaml --dry-run # authoring, not evidence
validshot render examples/ka-app/protocols/KA-001-order-form.yaml # review copy (PDF) for the reviewer
validshot lock examples/ka-app/protocols/KA-001-order-form.yaml --approver "A. Reviewer" --reason "Initial approval"
validshot run examples/ka-app/protocols/KA-001-order-form.yaml --initiator "T. Tester"
validshot verify-run runs/KA-001-<timestamp>-<id>
validshot report runs --title "Order form 1.4.2 — validation summary" # summary over all formal runs(validshot = node bin/validshot.mjs, or npm link once.)
Try the same locked protocol against the defective variant, which has four planted display defects:
validshot run examples/ka-app/protocols/KA-001-order-form.yaml --initiator "T. Tester" --base-url http://127.0.0.1:4173/badIt fails on exactly the planted defects: a total truncated to 10 cents, quantity 0
accepted, a stock badge covered by a promo banner, and a truncated product name. The
last two have correct DOM text. test/known-answer.test.mjs asserts this, and it is
the starting point for the tool's own OQ. The output of these commands is in
docs/sample/, with a table of which check caught which defect.
examples/stats-app/ is a worked conversion of a mock manual test script.
manual/TS-STAT-001_Batch-Statistics.xlsx is generated by manual/make_mock_xlsx.py;
the app, the company and the data are fictional. It becomes
protocols/STAT-001-batch-statistics.yaml. manual/CONVERSION.md maps every Excel
step to a protocol step and lists the decisions the reviewer must accept. One of them
is a wrong expected value in the Excel: it computed the population SD where the
requirement asks for the sample SD. Executed as written, the Excel would have passed
the defective app.
npm run stats-app # :4174, /good and /bad
validshot run examples/stats-app/protocols/STAT-001-batch-statistics.yaml --dry-runexamples/public-sites/ holds three protocols against real public sites. Each one
checks known answers that were computed or looked up independently of the site, plus
edge cases and response times (within_ms):
| protocol | site | checks |
|---|---|---|
| CALC-001 | calculator.net standard deviation | σ = 0.28284271247462 and s = 0.31622776601684 for the STAT-001 data set; small values; text input; single value; result within 5 s |
| REGEX-001 | regex101 | \d+ on a1b22c333 → 3 matches; no match; pattern error; catastrophic backtracking halted within 5 s |
| PDB-001 | rcsb.org | search 4HHB → entry listed; title, 1.74 Å, 2 unique chains (Fermi & Perutz 1984); results within 10 s |
validshot run examples/public-sites/REGEX-001-regex101.yaml --dry-runThese are demos, not validation. Public sites change without notice and expose no version. Respect each site's terms and run them occasionally, never in a loop.
A real one: docs/sample/runs/KA-001-20261110T154735Z-9a05/.
runs/KA-001-20261110T154735Z-9a05/
protocol.yaml exact bytes that were executed
protocol.lock.json who approved them, when, why
shots/step-01.raw.png as captured (read-only)
shots/step-01.annotated.png
record.json / .html / .pdf
audit.jsonl append-only, each entry hashes the previous one
manifest.json sha256 of every file + chain head
manifest.sha256 write this on the signed record / store it read-only
video/run.webm optional: the run as the browser recorded it (in the manifest)
manifest.sig.json optional: key signatures over the manifest
manifest.tsr (+ .json) optional: RFC 3161 timestamp token for the manifest
validshot verify-run re-hashes every file and walks the chain. A swapped
screenshot, an edited or deleted log entry, or a changed record is reported. If the
run is signed or timestamped, those are checked too.
A local hash chain detects an edit, but someone who can write to the run directory can rewrite all of it consistently. Two things over the sealed manifest close that gap, and both are optional:
validshot keygen keys/t-tester # Ed25519: t-tester.key.pem (private), t-tester.pub.pem
validshot lock p.yaml --approver "A. Reviewer" --reason "Initial approval" --sign-key keys/a-reviewer.key.pem
validshot run p.yaml --initiator "T. Tester" --sign-key keys/t-tester.key.pem --tsa http://timestamp.digicert.com
validshot sign runs/<run> --sign-key keys/a-reviewer.key.pem --signer "A. Reviewer" --role "Reviewed by"
validshot verify-run runs/<run> --trust keys/t-tester.pub.pem --tsa-ca /etc/ssl/cert.pem --require-timestamp- A key signature binds a signer's name, role and time to the manifest hash (or,
for a lock, to the protocol's bytes). With
--trust, verification fails unless a valid signature from that key is there, so a rewritten or unsigned run is rejected. Several people can sign the same run. - A trusted timestamp is a token from an RFC 3161 time-stamping authority saying
that this manifest existed at that time. Only the hash is sent. The run output and
manifest.tsr.jsonstate how far the host clock was from the TSA's time.--tsa-caverifies the token's issuer with OpenSSL; without it,verify-runonly confirms that the token is about this manifest. VALIDSHOT_SIGN_KEYandVALIDSHOT_TSAset the defaults for--sign-keyand--tsa.
These are not 21 CFR Part 11 electronic signatures. A key file proves that
someone held the key, not who that was. The private key is not encrypted. Assigning
keys to people, protecting and replacing them is your procedure. The gap is only
closed if the private key is out of reach of whoever can write to the evidence, and
if verification asks for the signature (--trust).
environment:
base_url: https://test.example.com/app
browser: webkit # chromium (default) | firefox | webkit | chrome | msedge
video: true # or `validshot run … --video` for one run- The engine is set in the protocol, so it is part of what the reviewer approves, and
it is on every record with its version.
chromium,firefoxandwebkitare the builds Playwright pins (npx playwright install firefox webkit).chromeandmsedgeuse the browser installed on the host, whose version validshot records but does not pin. Playwright's WebKit and Firefox are those engines, not the Safari and Firefox releases your users install. - The video is what the browser recorded of the test page: real time, not cut, no cursor or caption added. It is hashed into the manifest like every other file, and each step in the record lists its position in the video. It is supplementary: the verdicts rest on the raw step screenshots.
The workflow is: draft, dry-run, render, review, lock, run, report.
-
validshot render p.yamlwritesp.review.pdf, the protocol laid out like a manual test script, for reviewers who don't read YAML. It contains:- the requirements with their coverage, with gaps flagged;
- each step with its action in words and its expected results as numbered checks, with notes;
- the execution rules, a review checklist and signature fields;
- the exact YAML as an appendix.
The approved item is still the YAML file. Every page carries its sha256, and the sha256 the reviewer sees is the one
validshot lockpins. -
validshot report <runs-dir|run-dir>...writes a summary report over formal runs. It contains:- the results by protocol (most recent run) and requirement traceability;
- the full execution history, including failed and aborted runs;
- all deviations, with a column for the deviation-record reference;
- open items: earlier failures, uncovered requirements, overridden addresses, differing system versions and evidence-integrity failures.
Every run's evidence is re-verified when the report is generated, and dry runs are left out. The report draws no conclusion: section 7 is for the signatories. The report directory is sealed like a run, so
validshot verify-runworks on it.
The full YAML reference and edge-case checklist are in skills/validshot/SKILL.md. A step looks like this:
- id: 5
req: REQ-020
action: "Edge case, below range: enter quantity 0."
do: { fill: { target: { label: Quantity }, value: "0" } }
expect:
- { target: { role: alert }, text: Quantity must be a whole number from 1 to 99 }
- { target: { role: button, name: Place order }, enabled: false }validshot inspect <url> lists verified targets. With
--protocol p.yaml --after-step N it lists them for the state after replaying steps
1..N.
{ "mcpServers": { "validshot": { "command": "npx", "args": ["-y", "validshot@0.3.0", "mcp"] } } }Tools: validshot_guide, inspect_page, check_protocol, dry_run,
render_protocol. There is no tool for lock, formal run or report. For Claude Code,
also copy skills/validshot/ to .claude/skills/validshot/.
Done:
- Evidence Runner (M1), in Chromium, Firefox or WebKit, with an optional video;
- review copy, lock and summary report;
- key signatures and RFC 3161 timestamps over sealed evidence;
- display check on the rendered pixels as well as the DOM;
- agent authoring via MCP (M3; the 80 % acceptance metric has not been measured);
- known-answer apps and tests;
- a tool validation template pack in docs/tool-validation/: URS, risk assessment, IQ/OQ protocol based on known-answer testing, and known limitations.
Not done yet. Each of these needs to exist before validshot output can serve as formal evidence:
- Part 11 electronic signatures and access control (M4). Key signatures show which key signed, not which person; there is no authentication, and nothing is printed on the PDF. Records still carry wet-ink / external e-signature blocks, and the lock is a review record.
- WORM storage. A signature and a timestamp make a rewritten run detectable when verification demands them. They do not stop deletion, and they depend on where the private key is kept.
- Trusted time inside the record. The timestamp covers the sealed manifest. Step times come from the host clock; its distance from the TSA is recorded, NTP sync is not verified.
- PDF/A archival output, Word template import/fill-back (M2), traceability export to validation platforms (M6).
- An executed qualification. The validation pack is a set of templates; each
organisation executes the IQ/OQ in its own environment (
npm run oqwritesoq-results.tap). The known limitations list what the display check still cannot see. - Determinism means the same protocol, tool version and system state give the same steps and verdicts. Screenshots are not guaranteed pixel-identical.
npm test # unit, tamper detection, known-answer (good/bad apps), failure model,
# timing, display check, review copy, summary report, MCP surface,
# signatures, timestamps (needs openssl), video; Firefox and WebKit
# tests are skipped when those engines are not installed
npm run oq # the same suite as OQ evidence: also writes oq-results.tapMIT



