Roach is a recording HTTPS proxy for tests. It records chosen requests and replays them, so a second run of a test suite makes no live calls. The proxy owns every read and write of the recordings. A test runner only gives it rules and tells it when each test starts and ends.
Roach is a shared service (packages/service): one deployed proxy for many
projects and CI runs. A client points HTTPS_PROXY at it and trusts its
certificate authority. It installs nothing else. The recordings are in a
GCS bucket, which deletes each one 30 days after it was written. See
Service and Deploy.
Roach started in
the evals of
getsentry/junior.
It uses Node built-ins and the openssl command. The npm package has no
dependencies. Only the service uses the Sentry SDK (@sentry/node), and the
service is a private package in packages/service that is not published.
Install @sentry/roach, add a roach.json (see Config file),
and add two lines to the vitest config:
export default defineConfig({
test: {
globalSetup: ["@sentry/roach/vitest"],
setupFiles: ["@sentry/roach/vitest/setup"],
},
});The global setup (packages/roach/src/vitest.ts) starts one run for the invocation and
gives its proxy variables to the test workers. The setup file
(packages/roach/src/vitest-setup.ts) opens one session for each test, so test files
record in parallel workers at the same time. A passed test writes its new
recordings. A failed test drops them. In replay mode, a test with a miss
fails. At the end, the global setup writes the stats of the run to stdout
and to the job summary. After you add Roach, do the check in
Find changing values.
ROACH_TOKEN: the token of the service. Each run needs it. Only the global setup reads it. Without it, Roach is off, as withROACH_MODE=off: the tests run and their requests go direct. A local run needs no setup. WhenROACH_MODEasks for a run, such asreplay, the global setup fails without the token.ROACH_URL: the service. Default:https://roach-proxy.getsentry.net.ROACH_MODE:auto,replay,record, oroff. Default:auto. Setreplayfor a job that must not record, such as a check that all tests replay. Withoff, the global setup starts no run, so requests go direct and need no service and no token.
A project that has its own global setup calls startRoach(project) from
@sentry/roach/vitest in it, and returns its teardown.
- It needs Node 24.14 or later (
http.setGlobalProxyFromEnv), except inoffmode. - Put
@sentry/roach/vitest/setupbefore a setup file that starts an HTTP mock, such as MSW or nock. The setup file keeps the request functions ofnode:httpandnode:httpsas it loads, so its calls to the service do not go to the mock. - The tests of one file share one process, so they run one at a time. Do
not use
test.concurrent. - The setup file sets a new global
fetchdispatcher for each test. A dispatcher that you set before, such as one withcompose()interceptors, does not apply in the test. Set it again in abeforeEachof a setup file that comes after@sentry/roach/vitest/setupinsetupFiles, withsequence.setupFiles: "list". Restore it withonTestFinished. - A client that gives
fetchits own undici dispatcher, such as@vercel/sandbox, does not use the proxy. Its requests go direct. - The package has TypeScript sources. Vitest compiles them, but Node does
not strip types in
node_modules, so plain Node scripts cannot import it.
import { startRemoteRun, connectRoach } from "@sentry/roach/client";
import { describeRecordingStats } from "@sentry/roach/report";
import { VALUE_PATTERNS } from "@sentry/roach/values";
// Once per CI job. Each run needs the token of the service.
const run = await startRemoteRun(
{ url: "https://roach.example.com", token: process.env.ROACH_TOKEN! },
{
tenant: process.env.GITHUB_REPOSITORY!, // such as getsentry/junior
mode: "auto",
name: `${process.env.GITHUB_RUN_ID}-${process.env.GITHUB_RUN_ATTEMPT}`,
rules: [
{
name: "model",
match: { method: "POST", url: "https://ai-gateway.vercel.sh/" },
keyHeaders: ["ai-model-id"],
values: { uuid: VALUE_PATTERNS.uuid, time: VALUE_PATTERNS.isoTime },
},
],
},
);
spawn("vitest", { env: { ...process.env, ...run.env } });
// Once per test, in a worker that has the `controlUrl` and `token` of the
// run. Send the requests of the test through `session.url`.
const session = await connectRoach({ controlUrl, token }).startSession(
testName,
);
const { missed } = await session.end(passed);
// In `replay` mode, fail the test when `missed` is not 0.
// At the end of the job.
console.log(describeRecordingStats(await run.close()));run.env has HTTP_PROXY, HTTPS_PROXY, NO_PROXY, NODE_USE_ENV_PROXY,
and NODE_EXTRA_CA_CERTS. Node 24 reads the last two only at startup. The
CA file also has the certificates of NODE_EXTRA_CA_CERTS of the caller.
A client that does not use client.ts can do the same with plain HTTP. See
Control API.
A repository adds one step and a roach.json file (see
Config file):
- uses: getsentry/roach@v0
with:
token: ${{ secrets.ROACH_TOKEN }}
run: pnpm testThe action (packages/roach/src/action.ts) starts a run with the token in
auto mode. Set ROACH_MODE: replay in the env of the step for a job
that must not record. Without the token, it runs run without Roach, so
its requests go direct, and the job summary says so. With the token, it
runs run in
bash with the proxy variables
of the run, then ends the run and writes its stats to the job summary. The
url input sets the service. Default: https://roach-proxy.getsentry.net.
- Only
runuses the proxy. Later steps, such as cache and artifact uploads, do not.rundoes not get the token or theACTIONS_*variables of the runner, such asACTIONS_RUNTIME_TOKENandACTIONS_ID_TOKEN_REQUEST_TOKEN. Soruncannot get an OIDC token. runis one session. When it fails, the run drops its new recordings. A miss inreplaymode fails the step. The summary names the job, not the test, of each miss. For a session per test, use vitest inrun.NODE_EXTRA_CA_CERTSworks only for Node. Python andcurlneed a CA bundle that also has the system certificates.
The action and the vitest integration read roach.json in the working
directory or the closest parent directory (packages/roach/src/config.ts):
{
"allow": ["https://api.anthropic.com", "https://openrouter.ai"],
"rules": [
{
"name": "model",
"match": {
"method": "POST",
"url": [
"https://api.anthropic.com/v1/",
"https://openrouter.ai/api/v1/"
]
},
"values": {
"uuid": { "builtin": "uuid" },
"build": "build-[0-9]+"
}
}
]
}allowandrulesare those of the run. See Config.- A value is a regular expression source, or
{ "builtin": "<name>" }for a pattern ofVALUE_PATTERNSinvalues.ts. secretEnv: environment variables that hold credentials that no request header has, such as a token that goes to a model in a prompt. Their values are thesecretsof the run. Do not list API keys that go in a header: the proxy redacts those on its own. See Config.tenant: defaultGITHUB_REPOSITORY. Set it, so local runs use the recordings of CI.noProxy: hosts that do not use the proxy. Default: loopback.
The name of the run is <GITHUB_RUN_ID>-<GITHUB_RUN_ATTEMPT> in GitHub
Actions. The run also sends ci: GITHUB_RUN_ID, GITHUB_JOB,
GITHUB_REPOSITORY, and GITHUB_REF. Only the spans in Sentry use it.
A run takes RunConfig (packages/roach/src/types.ts): tenant, mode, rules,
allow, name, secrets, and ci.
mode:autoreplays recordings and records misses.replayreplays recordings and fails a miss with HTTP 412, so nothing goes live.recordsends every request live and records it again.offrecords and replays nothing.allow: the only origins that the proxy sends requests to, 1 to 32 of them. It refuses all other origins with HTTP 403. The upstream host always comes from this list, not from the request. The service sends requests only to public addresses, so an origin that resolves to a private, loopback, or link-local address also gets HTTP 403. So does an IPv6 address with an IPv4 address in it, such as[::ffff:808:808]or[::ffff:0:7f00:1]. The stats of the run count the refused requests by origin, inrefused.notAllowedandrefused.notPublic. The run report shows them afterRefused:, so look there first when requests fail with HTTP 403. A tunnel to a port other than 443 shows ashost:port. The stats name 32 origins for each reason, and count the others asother origins.rules: the traffic to record.namestarts with a letter or a digit, has at most 64 characters, and uses letters, digits,_,., and-.matchhasmethod,url, andheaders.urlis a prefix, or a list of 1 to 32 prefixes. A URL that starts with one of them matches. Use a list when the same API is on many hosts, such as one rule with the samevaluesfor Anthropic, OpenRouter, and OpenAI. The key does not usematch, so a change tourlkeeps the keys. When you merge rules into one rule, some keys get a new rule name, so their requests record again one time.keyHeadersadds request headers to the key.valuesnames the values that change from run to run. A run can have 64 rules, and a rule can have 32 values.- Credentials: the proxy never writes the
secretsof the service config or of the run. Usesecretsof the run for the credentials of a CI job that a response can show but no request header has, such as a token that goes to a model in a prompt. The service keeps them only in memory, for the run. A run can have 32 of them, each of at most 16 Ki characters. It also learns the value of each request header and query parameter whose name can mean a credential: a name withauth,token,secret,key,cookie,session,password,credential, orsignature. The match is broad on purpose. It does not learnx-session-affinity, because that header has a conversation id and not a credential. Recordings show<<redacted>>in place of each known value. The URL of a recording, a request sample, and the text of a miss in Sentry also show<<redacted>>for the value of each such query parameter, at any length (see Misses). They also show it for a parameter whose value is the value of a credential field of the request body (see below). The other parameters do not change. A run learns at most 16,384 values, with a total of 16 Mi characters, for credentials that change on each request, such as AWS signatures. The whole value and each part of it count, also for thesecretsof the run. The proxy also redacts each part with\/for/, as some servers write it in JSON, and the whole value when a/is only in parts shorter than 12 characters. These forms do not count. A run whosesecretsgo past the limit gets HTTP 400 when it starts. Past the limit, a request with a new credential gets HTTP 431, and the proxy does not send or record it. A request with known credentials goes on. Start a new run to learn more. - Compressed bodies: the proxy asks the upstream for a body without a
content encoding. When the upstream sends
gzip,deflate,br, orzstdall the same, the proxy decodes the body before it redacts and records it. It decodes 64 MiB at most. When the body does not decode, or when it decodes to more than 64 MiB, the proxy does not record it. It writes a log line, and the client gets the body as it came. - Bodies that are not text: a recording keeps such a body, such as
application/octet-stream, as base64. The proxy redacts the known values in its bytes before it encodes it. Only the bytes of a known value change, so a binary body stays binary. The client gets the same bytes as a replay gives. - Credentials that the upstream makes: in a response body, the proxy also
redacts the value of each field that holds a credential that a server
mints, such as the token of an OAuth server. In JSON, only the fields
of the object at the top count, because a server puts a minted
credential there. In an
application/x-www-form-urlencodedbody, the form fields count. The fields areaccess_token,refresh_token,id_token,session_token,auth_token, andclient_secret. A baretokenfield counts only when its value starts like a credential: a GitHub token (ghp_,gho_,ghu_,ghs_,ghr_, orgithub_pat_), a JWT (eyJand two dots), or a Slack token (xoxa-,xoxb-,xoxp-,xoxr-, orxoxs-). Model APIs usetokenfor text, such as in log probabilities, so the length alone never counts. The names compare without case,_, and-, soaccessTokenis the same asaccess_token.- The list is short, and only the top of a JSON body counts, on
purpose. A model can repeat a part of its response in a later
request, such as the input of a tool call. A redacted value there
gives a new key, so the request misses on replay. So the proxy does
not redact a field such as
passwordin a response, or a field at a lower depth, or text in a string, such as?access_token=in a command. - A recording shows
<<redacted>>in place of the value. The proxy replaces only the value after the name of its field. All other text of the body keeps its bytes, also the same text at other places, and the spaces and the order of the fields. So a JSON body stays valid JSON. A byte order mark at the start and unicode escapes in the value do not stop it. The proxy also does this for a body that is valid UTF-8 JSON when its content type is not JSON, or when it has none. When a minted value stays in the body after this, the proxy does not record the response. It writes a log line, and the client gets the body as it came. - The proxy does not find these fields in text that is not one JSON
value, such as the
data:lines of a stream (text/event-stream), or JSON that is not valid. It keeps them there. It still redacts each known value in such a body. When a field name of the top object comes two times, it reads only the last value, asJSON.parsedoes. A recording keeps anapplication/x-www-form-urlencodedbody as text. Thelocationheader of a recording shows<<redacted>>for each known value, and for each of these fields in its query and its fragment, such as#access_token=. Other parameters do not change, such askeywordorpageToken, so a replay can follow the redirect. - A live response gives the client the real value, so the run can use
the credential in its next requests. From then on, the run also
redacts that value where it knows credentials, when it is 12
characters or longer. These values do not count toward the limit
above, because only a run that records gets them, from an upstream
that
allownames. - A replay gives
<<redacted>>in place of the value. A client that only sends the credential back in a header, such asAuthorization, replays without a change, because the key does not use that header. A client that reads the credential, such as one that decodes a JWTid_token, or that sends it in a body, in the URL query, or in a key header, does not work on replay. Do not record such requests with a rule.
- The list is short, and only the top of a JSON body counts, on
purpose. A model can repeat a part of its response in a later
request, such as the input of a tool call. A redacted value there
gives a new key, so the request misses on replay. So the proxy does
not redact a field such as
- Credentials in a request body: a request sample and the text of a miss
in Sentry show
<<redacted>>for the fields above, and also forbearer_token,client_assertion,password,api_key,secret_key,private_key, andsecret_access_key. In JSON, these fields count at each depth. In anapplication/x-www-form-urlencodedbody, the form fields count, also with a nested name such asuser[password],password[], oruser[password][0]. No key uses this text, so this list can be longer. When the value of such a JSON field is an object or an array, each string in it shows<<redacted>>, at any depth. When the value is a number, or has a number in it, the request sample has no text, and the URL of the recording shows<<redacted>>for a parameter with that number. The proxy replaces the value at the place of each field, and each other string that is the same value, also each string of an object or an array value. A value pattern can change the text of a key, so then the proxy checks the result. It parses the body again, and the body must be JSON. Each field must hold<<redacted>>at its place, and no string can be one of the values. Only the body counts, not a key header. When a check fails, the request sample has no text, and the proxy writes a log line with the key. This occurs when a value pattern takes a quote, or a name of a field, away. A request with credential fields also gets no text when the name of a value of its rule has",\,&,=,<,>, or a control character, because its<<name>>can add a field. A miss has no change then. - The proxy cannot find the fields of a body that is not a JSON object or
array or a form, such as JSON with a comma at the end, JSON lines, a
multipart body, or a JSON string. It also cannot find them in a form body
that has no credential field, such as JSON or XML with a form content
type. So the request sample of such a body has no text when the body has
the name of a field above anywhere, or the start of a bare
tokenabove. The name can be in any case, with_,-, or nothing between its words, such asaccessToken, and with JSON unicode escapes or URL escapes in it. Each test is one search of the body, so a long body costs little. In a form with credential fields, the request sample also has no text when a field that is not a credential field can show one, such asuser.password=, or a baretokenor another field with JSON with URL escapes in its value. A value that is only such a name, such asgrant_type=password, does not count. It also has no text when a part of a nested name, such asSofS[password], is a credential value, or when a name has the start of a baretoken. - The proxy parses a JSON body to find the fields, so a long string value cannot stop it. It does not find a field in JSON text that is itself a string, such as the escaped arguments of a tool call, or a copy of a value inside other text, such as in a prompt or with URL escapes. The URL of a recording keeps a copy of a value of the body when the proxy cannot find the fields of the body.
- Keys: redaction never changes the key of a request. The key is the hash of the request as the client sent it, so recordings from before stay valid.
Requests that match no rule go live without a change, and their responses
stream. Each response has an x-roach header: replayed, live,
missed, or passthrough.
The key of a recording is <rule>/<hash>.json, under <tenant>/ in the
store. The hash is the hash of the rule name, the method, the URL, the key
headers, and the body. A JSON body has sorted keys. A recording keeps the
response, the session that recorded it, and the method and the redacted
URL of the request. Of the response headers, it keeps only content-type
and location. It does not keep the request body. A request sample keeps
it. See Misses.
- Many sessions can be open at once. A request belongs to the session of
its proxy URL (
urlof the session). A request through the proxy URL of the run belongs to the open session when only one is open. The proxy keeps the new recordings of a session in memory. A passed session writes them, and a failed session drops them, so a bad sample is never replayed. - A request belongs to the session that was open when it started. If it ends after its session ended, it follows the result of that session.
- A request outside a session is written at once.
- A 429 or 5xx response is never recorded. A replay does not write the file again.
Ids and times change on each run, and a response often repeats them. For
example, the model archives the memory with the id that its request
showed. values of a rule maps a name to a regular expression source.
VALUE_PATTERNS in packages/roach/src/values.ts has uuid, isoTime, date, epochMs,
sha256, and gitCommit. When two patterns match at the same place, the first one wins.
The proxy matches values in JSON text. Start a custom pattern with
NOT_AFTER_WORD, not \b: after an escape such as \n, \b sees the n
as part of a word and does not match.
- The key sees each value as
<<name>>. So two runs with other ids or times use the same recording. - When the proxy records a response, it writes each value of the request as
<<name:n>>: the n-th value of that name in the request. - On replay, the proxy writes the n-th value of the current request there. A replayed response thus uses the ids of this run.
- A value that the response makes itself, such as a date that a model calculates, stays as it was recorded.
- A session remembers the values of its requests. In a later request of
the same session, such a value is also a placeholder where no pattern
finds it. For example, a pattern finds a short commit id only in the
output of
git push, but the model then quotes it alone in its reply. A pattern wins at the same place. A session remembers all values of its requests until it ends. It does not drop a value, because that would change the keys of later requests. Only a value of over 32,767 characters stays out: V8 cannot run a pattern with it, so it would fail each later request of the session. - Values in thinking blocks of the request do not count for
n, and the proxy does not look for remembered values in them. A replay keeps the recorded text of a thinking block, so when a later request sends it back, its values come from the recording run.
Model streams send a tool call in many small deltas, so a value can be
split over two events. Before it records a text/event-stream, the proxy
merges the deltas of each Anthropic Messages content block into one event.
Thinking blocks keep their recorded text, because their signature covers
it, and the provider refuses a changed thinking block.
Keep the patterns narrow. A broad pattern makes requests that differ in a real way use the same recording.
The service takes any pattern that compiles in 1 s, up to 2048
characters. A replay run must use the same patterns as the run that
recorded, or its keys do not match. Each key has a time limit: 1 s,
and 0.5 s for each MiB of the body. The built-in patterns take about 60
to 110 ms for each MiB. A request whose key takes longer fails with HTTP 400.
So does a JSON body with arrays or objects nested over 64 levels deep.
All runs share one process, and the service makes each key in it. Some
work, such as JSON.parse() of a large body, cannot stop at the time
limit. The key of a 64 MiB body can need about 1 GiB. Only CI jobs with
the token can start runs, so the service trusts their patterns and bodies
up to these limits. The limits never change a key: a request gets the key
that it always had, or it fails.
Avoid nested quantifiers, such as (a+)+.
The proxy adds each request without a recording to stats().misses, with
its session and key. The stats keep the first 500 misses of a run.
describeRecordingMisses() in packages/roach/src/report.ts gives the
first miss of each test, which is the one to fix. The run report of the
vitest integration and of the GitHub Action shows these misses.
The proxy also finds where the first miss of each session changed
(packages/service/src/misses.ts). Sentry gets the text of the change, on the roach.request
span of the miss (see Sentry). The report of a run tells only
where the change is:
- a.fixture.ts > passes: model/074d….json
Changed at character 65 of the request of model/8b96….json. See miss.recorded and miss.request on the roach.request span in Sentry.
The report goes to CI logs and job summaries, and these can be public. So no report, log, or stats of a run show the text of a request. Only Sentry shows it.
- When the proxy records a request, it also writes a request sample to
_requests/<rule>/<hash>.json. The sample has the text that the key hashes: the rule, the method, the URL, the key headers, and the body with<<name>>for each changing value. The proxy redacts credentials in it, as in a recording, the value of each credential query parameter, and the value of each credential field of the body, such asclient_secret(see Config). It keeps the first 1,048,576 characters (UTF-16 code units) of the text. - The hash of a sample is the place of the request: its session (test) and the number of requests of the rule that the session sent before it. So a sample is the last recorded request at the same place of the same test. A replay writes no sample.
- At the first miss of a session, the proxy reads the sample of the place of the request. It finds the first character that is not the same. Sentry gets 40 characters before it, and at most 120 characters from it, of each text. This is one read for each session, in every run.
- The stats and report of a run get
change. It has the key of the recording that the sample came from (file), and the offset (at). It has no text. - A miss has no change when there is no sample, such as for a recording
from an older version, when the sample has no text (
withheld, see Config), or when the texts are the same in the part that the sample keeps. A sample with no text still takes the place of the older sample, so a miss never compares with a request that is not the last. - The proxy redacts the text of the miss before it compares it. This has the time limit of a key (see Changing values). Past the limit, the miss has no change.
- When the requests of a test run in parallel, their order can change. Then a sample can come from another request of the test.
Do this check when you add Roach to a project. Do it again when a test that passed does not replay on the next run.
A value that changes on each run and that no pattern finds gives a new key
on each run. In auto mode, that request goes live and records again on
each run. The run is slow, and it costs as much as a run without Roach.
The requests after it in the test also go live, because the live response
is not the same as the recorded one. No error shows this. The tests pass.
-
Run the suite two times with the token. On the second run, each test that passed must replay all of its requests. Look at
livein the stats of the second run. It must be 0, or near 0. In CI, also compare the time of each test. A test that replays takes about as long as its code without the network.TELEMETRY.mdshows how to find the runs and keys that stay live. -
Find the first live request of a test that did not replay. Run only that test with
ROACH_MODE=replay. The test fails at the first miss, and the run report shows its key. Fix the first miss of a test before the others. The others often come from it. -
Find the
roach.requestspan of the first miss in Sentry, by itsroach.run.id,roach.session, andhas:miss.changed_at. Readmiss.recordedandmiss.request. See Misses andTELEMETRY.md. They show the changing value, with the text before it. When the miss has no change, get the body of that request from two runs, and compare them. For example, add this file tosetupFilesafter@sentry/roach/vitest/setup, and run the test two times:// Local only. Writes each fetch request to DUMP_FILE. import { appendFileSync } from "node:fs"; import { expect } from "vitest"; const dumpFile = process.env.DUMP_FILE ?? "requests.jsonl"; const fetchBefore = globalThis.fetch; globalThis.fetch = async (input, init) => { const request = new Request(input, init); const record = { test: expect.getState().currentTestName, method: request.method, url: request.url, body: await request.clone().text(), }; appendFileSync(dumpFile, JSON.stringify(record) + "\n"); return fetchBefore(request); };
The first difference between the two bodies is the changing value. This file sees only
fetch. A client that usesnode:httpdirectly needs its own hook. -
Add a narrow pattern for the value to
valuesof the rule. See Changing values. Then do step 1 again.
Values that changed in real projects:
- Ids that the code under test makes, such as UUIDs and random short ids.
For example, Warden gives each finding a random
XXX-XXXid, and the next model request quotes it. - Paths of temporary directories, commit SHAs of a repository that a test makes, and times and dates.
- The local environment. Global git config, such as a hook that adds a
commit trailer, commit signing, or
user.name, changes the commits that a test makes.HOME, the user name, the locale, and the time zone can also show in a request. Then local keys are not the same as CI keys, so a localreplaymisses while CI replays. Make the test harness ignore them. For git, setGIT_CONFIG_GLOBAL=/dev/nullandGIT_CONFIG_NOSYSTEM=1, and set the author and committer in the test. - Order. The proxy sorts JSON object keys, but not arrays. An array whose order depends on timing, such as the results of parallel tool calls, gives a new key when the order changes.
packages/service/src/cli.ts service <config.json> starts the service. The
config is RoachServiceConfig in packages/service/src/service.ts. deploy/gcp/ writes it for you.
- Tenants. A tenant is a GitHub repository, as
owner/repo, such asgetsentry/junior. In GitHub Actions, useGITHUB_REPOSITORY. The service has no list of tenants, so a new project needs no change to the service. The recordings of a tenant are under<owner>/<repo>/in the bucket. Tenants never share recordings. - One token. Each run needs the token, in every mode, to read or
write recordings. All tenants share it. The config keeps only its
SHA-256, in
writeTokenHash. The tenant name is only a label, not proof of identity: a job with the token can read and write the recordings of any tenant. So give the token only to repositories and workflows that you trust, and never to a workflow that runs code from a fork. CI jobs of forks get no secrets, so Roach is off in them, and their requests go direct. - Runs. Each run has its own id, token, mode, rules, sessions, and
stats. A proxied request names its run in
Proxy-Authorization, asBasic base64(<id>:<token>). The proxy URL of the run has them, soHTTPS_PROXYsends them. A request without them gets HTTP 407. - Config of a run. Each run sends its own
allowandrules, with their value patterns. A tenant that calls a new origin or uses a new pattern changes only its own config. The service needs no change and no restart. Two checks keep one run from harming the others:- The proxy sends requests only to public addresses
(
packages/service/src/addresses.ts). It checks the address of each connection after DNS, so a host name cannot point a run at the metadata server, at the VPC, or at the VM. Only tests and local development setprivateNetworks. - A key has a time limit. So a slow pattern fails only its own request.
- The proxy sends requests only to public addresses
(
- Sentry. With
sentryDsn, each read and write of a recording sends a span to Sentry. Each proxied request sends a span and metrics. Each unexpected error sends an error event. See Sentry. - Errors. An error that is the fault of the client, such as HTTP 400,
401, 409, 413, or 429, keeps its message. Any other error, such as a failure
of the upstream or of GCS, gets HTTP 502 for a proxied request or HTTP
500 for a control call. Its text is only
internal error, Sentry event <id>, because the message of the error can have a response body. Only Sentry gets the error. The service log gets its class, its code, and the HTTP status of a failed GCS call. - Certificate authority.
caholds the PEM certificate and key that clients trust. Keep it across restarts. Host certificates live for 7 days, and the service makes new ones before they expire.
With sentryDsn, the service sends metrics, spans, and unexpected errors
to Sentry. It keeps all traces. TELEMETRY.md has the queries. Each
attribute of Roach starts with roach.. Metrics have only attributes with
few values: roach.tenant, roach.mode, roach.rule, roach.origin,
roach.result, and roach.reason.
| Metric | Type | Value |
|---|---|---|
roach.request |
counter | 1 for each proxied request. roach.result is replayed, live, missed, passthrough, refused, or aborted (the client closed the request before its body came). A passthrough, refused, or aborted request has no roach.rule. A refused request has roach.reason, and other for an origin not in allow. |
roach.request.duration |
distribution (ms) | The time to answer a request that a rule records, by roach.result. |
roach.recording.write |
counter | Recordings by roach.rule. roach.result is written (the proxy sent them to the store) or dropped (their session failed). |
Each proxied request is one trace. Each span has roach.tenant,
roach.mode, roach.run.id (the name of the run, else its id), and the
roach.ci.* attributes of the CI job of the run: roach.ci.run_id,
roach.ci.job, roach.ci.repo, and roach.ci.ref.
roach.run.start: a new run.roach.request: one proxied request, named<method> <origin>, withroach.origin,roach.rule,roach.result,roach.session(the name of the session, such as the test),http.request.method, andhttp.response.status_code. A refused request also hasroach.reason. When a replayed or live response has token usage, the span also has thegen_aiattributes of the model call (packages/service/src/usage.ts). Sentry then calculates its cost in USD asgen_ai.cost.total_tokens. The span of the first miss of a session also hasmiss.sample(found,withheld, ornone), in every run. When the proxy finds a change, the span also hasmiss.changed_at(the offset of the first different character),miss.compared(the key of the recording), and the redacted text of the change:miss.recordedandmiss.request. Only Sentry gets these texts (Misses). It contains:roach.key: the key of the request, withroach.ruleandroach.key.roach.store.read: a read of a recording, withroach.keyandroach.result:replayed(the store has the recording) ormissed. The first miss of a session also reads a request sample. Itsroach.keystarts with_requests/.roach.upstream: the live request, withroach.origin,http.request.method, andhttp.response.status_code.
roach.store.write: a write of recordings, withroach.recordingsandroach.written. A session writes when it ends, so this span is usually its own trace. A second span writes the request samples of the same recordings.roach.run.end: the end of a run, with its totals:roach.replayed,roach.live,roach.missed,roach.passthrough,roach.refused,roach.written,roach.dropped, androach.run.duration_ms.
An error event has the tags of the spans of its run. An error of the control API has no run tags.
Telemetry never has a credential. It has no body and no path or query of
a URL, except miss.recorded and miss.request. These have at most 160
characters of the redacted text of a request, which can include a part of
its URL, headers, or body. The log of the service names only the key of a
miss, not its change.
From a model call, it has only the model name and the token counts.
So the service turns off the request data and the HTTP spans of the SDK,
removes the path from the name of an error event, and adds no trace
headers to the requests that it sends. An error event has the message of
the error, which can have a response body.
TELEMETRY.md shows how to get the hit rate, the misses,
the time that replays save, the dropped recordings, and the keys that
change on each run.
- Runs live in memory in one process. A restart ends the open runs, and their CI jobs fail. A run that is open for 6 hours ends as failed.
- A proxied request body over 64 MiB gets HTTP 413. A control API body over 1 MiB gets HTTP 413 before the service parses it. A start of a run without the token gets HTTP 401 before the service reads its body.
- With
proxyProtocol, the service reads and drops the PROXY protocol v1 header of the load balancer. A connection without the header works too. - Keys and the redaction of misses have time limits. See Changing values.
- Request headers that do not come in 10 s get HTTP 408, and a request
whose body does not come in 300 s too. This is also true in a tunnel
and with
proxyProtocol. WithproxyProtocol, a connection that sends no byte, or no full PROXY header, in 10 s closes. - A
CONNECTtunnel that sends no byte for 10 s after the200closes. - A connection or tunnel that sends and gets no byte for 10 minutes
closes, also while a request waits for its response. This is true for
each request, with or without TLS. Between two requests, a connection
closes after 6 s without a new request (the
keepAliveTimeoutof Node, 5 s, and 1 s that Node adds), and the client opens a new one. - The credentials that a run learns have a limit. See Config.
- GCS deletes each recording 30 days after it was written, also when runs
still replay it. Then
automode records it again, andreplaymode fails the request with HTTP 412. - A write replaces the recording with the same key.
deploy/gcp/ is the production setup, in Terraform. It makes:
- a GCS bucket that deletes each recording after
recording_days(30), - the certificate authority and the token,
- the service config, in Secret Manager,
- one Container-Optimized OS VM that runs the image
ghcr.io/getsentry/roach:main, with a service account that can only use the bucket and the secret, - an SSL proxy load balancer with a Google-managed certificate. It ends TLS
and passes the raw TCP stream, with
CONNECT, to the VM. Each connection starts with a PROXY protocol v1 header with the address of the client (proxy_header). The config setsproxyProtocol, so the service reads and drops it. The health check sends the header too, - a service account for the
Deployworkflow, which GitHub Actions uses through workload identity federation (deploy.tf). It can only reset the VM.
The Image workflow builds the image on each pull request and pushes it as
:<sha> on main. The VM pulls :main at each start.
The Deploy workflow deploys main when it is green. After the Image
workflow of a commit on main, it waits for CI of the same commit. When
both pass and the commit is still the head of main, it moves :main to
:<sha>, resets the VM, and waits until the service answers. A reset ends
the open runs, so their CI jobs fail. A red commit does not deploy, and an
older commit never replaces a newer one.
Sometimes GitHub starts no workflow for a push to main. Then the head of
main has no CI and no image, and it does not deploy. Run both by hand:
gh workflow run ci.yml --repo getsentry/roach --ref main
gh workflow run image.yml --repo getsentry/roach --ref mainThe Deploy workflow accepts these runs like push runs.
-
Make the package
ghcr.io/getsentry/roachpublic once, after the first push tomain, so the VM can pull it without credentials. -
deploy/gcp/roach.tfvarshas the values of the Sentry deployment: the GCP projectroach-511216, the domain, and the DSN of thesentry/roachproject. Another deployment changes these values. It has no origins or patterns: runs send them. -
The Terraform state is in the private bucket that
backend "gcs"inversions.tfnames. The state holds the CA key and the token, so the bucket must stay private. Make the bucket before the firstterraform init. -
Log in with
gcloud auth application-default login, then apply:cd deploy/gcp terraform init terraform apply -var-file=roach.tfvars -
Point an A record of
domainat theip_addressoutput. The certificate works some minutes after DNS does. -
Put the token in one GitHub organization secret,
ROACH_TOKEN. Give it access only to the repositories that you trust. This command does not show the token:terraform output -raw write_token \ | gh secret set ROACH_TOKEN --org getsentry --visibility selected \ --repos getsentry/junior
-
Check it:
curl https://<domain>/__roach/ca.pemreturns the CA certificate. -
Let the
Deployworkflow reset the VM. Set two variables of the repository from the outputs. Until they are set, the workflow only moves:main, and the VM gets it at its next start:gh variable set GCP_WORKLOAD_IDENTITY_PROVIDER --repo getsentry/roach \ --body "$(terraform output -raw deploy_workload_identity_provider)" gh variable set GCP_DEPLOY_SERVICE_ACCOUNT --repo getsentry/roach \ --body "$(terraform output -raw deploy_service_account)"
Another deployment also changes
deploy_repositoryanddeploy_repository_idin its tfvars, and theenvof.github/workflows/deploy.yml.
To deploy by hand, restart the VM:
gcloud compute instances reset roach --zone <zone>. It pulls :main,
the last green commit.
The service reads its config only when it starts. When a change to the
service config must reach the VM before a change to the load balancer,
such as proxyProtocol before proxy_header, apply in this order. Run
the commands in deploy/gcp:
-
Merge the change, and let the
Deployworkflow deploy the image. -
Apply only the config secret:
terraform apply -var-file=roach.tfvars \ -target=google_secret_manager_secret_version.config
-
Reset the VM, so that it reads the new config:
gcloud compute instances reset roach --project roach-511216 --zone us-east1-b
-
When the service answers, apply the rest:
terraform apply -var-file=roach.tfvars
Step 1 and step 3 each reset the VM, and a reset ends the open runs, so their CI jobs fail. Do step 3 when few runs are open, and do step 4 at once after it.
With proxyProtocol, the service reads a PROXY header when a connection
has one, and it uses the address of the connection when it does not. The
load balancer writes its header before the first byte of the client, so a
client cannot put its own header first. A client can send its own header
only when the service has proxyProtocol and the load balancer does not
send the header: from step 3 to step 4 above. So do step 4 at once. A
false header changes only the address for the limit of runs for each
client, and the firewall lets only the Google ranges connect to the VM.
To add a project, add its repository to the repositories of the
ROACH_TOKEN organization secret, in the GitHub settings of the
organization. Its CI jobs set tenant to GITHUB_REPOSITORY, and send
their own origins and patterns. The service needs no change.
To change the token, run
terraform apply -var-file=roach.tfvars -replace=random_password.write_token,
set the secret again, and restart the VM.
node packages/service/src/cli.ts service <config.json>It starts the service and prints its URL as one JSON line.
The calls of a run are under /__roach/runs/<id>. They need
Authorization: Bearer <run token>. client.ts calls them.
POST /sessionwith{"name": "..."}: open a session. Returns{"url": "..."}, the proxy URL of the session. Its user is<run id>-<session id>. An open session with the same name fails first. A run can have 64 open sessions. Another one gets HTTP 429. A service before 0.2.0 returns no URL. Thenclient.tsfails and tells you to deploy a newer service.POST /session/endwith{"name": "...", "passed": true}: end it. Returns{"missed": 0}, the misses thatreplaymode failed. Returns HTTP 409 when the session is not open, and HTTP 500 when the store fails (see "Errors" in Service).GET /stats: the totals of the run.
The service also has:
POST /__roach/runswith aRunConfig: start a run. SendAuthorization: Bearer <token>of the service, in every mode. Returns HTTP 201 with theid,token, proxyurl,controlUrl, andcaCertof the run. A call without the token, or with a wrong one, gets HTTP 401 before the service reads the body. A body that is not JSON or a config that is not valid gets HTTP 400, and a body over 1 MiB gets HTTP 413.DELETE /__roach/runs/<id>: end the run. Returns its stats.GET /__roach/ca.pem: the CA certificate. No token.
A call of a run with a wrong token gets HTTP 404, as a call of a run that does not exist does.
Run the Release workflow. Choose a bump: auto (the default), pre,
patch, minor, or major. auto picks the bump from the commits. pre
makes a prerelease of the version that auto picks. To release an exact
version, type it in version. Craft (.craft.yml) makes a
release/<version> branch and asks for approval in
getsentry/publish. When a release
manager accepts it, Craft:
- copies the image
ghcr.io/getsentry/roach:<sha>to:<version>. - makes the GitHub release
v<version>, with the changelog, and moves thev<major>tag. When the changelog has more than 5 entries, Craft adds a short AI summary above them. Repositories use the action asgetsentry/roach@v<major>. - publishes
@sentry/roachto npm: the tarball that thepackagejob ofCIuploads for the release commit. Craft publishes onlysentry-roach-<version>.tgz, and CI makes sure that it hasLICENSEandREADME.md(.github/scripts/check-tarball.sh). On the release branch, Craft runs.github/scripts/bump-version.sh, which writes the version topackages/roach/package.json. The workspace root andpackages/serviceare private and have no version.
The action runs packages/roach/src/action.ts from the tag, and the
package has the sources, so a release has no build step.
Craft takes the last version from the newest v<version> tag in the
history of main. Do not rewrite main, or the tag leaves its history and
Craft bumps from 0.0.0. The workflow deletes the floating v<major> tags in
its checkout first (.github/scripts/drop-floating-tags.sh), because
git describe picks them over the release tag on the same commit.
The Release workflow uses the token of Sentry Release Bot. A push with
GITHUB_TOKEN starts no workflows, so the release branch gets no checks and
no image, and Craft cannot publish it.
pnpm install
pnpm check # format, lint, types, unused code, and testsCI runs the tests only with coverage (pnpm test:coverage), and packs
@sentry/roach. It does not run pnpm check, so run it before you push. AGENTS.md has the conventions and the commands for one
file.
packages/roach/tests/package.test.tschecks what the npm package loads and ships.packages/service/tests/recording.test.tstests modes, sessions, changing values, and misses.packages/service/tests/redaction.test.tstests the redaction of credentials in recordings and request samples.packages/service/tests/service.test.tstests the service in the test process: access, limits, and the control API.packages/service/tests/vitest.test.tsruns vitest with the integration on a small project, against a service in the test process.packages/service/tests/deployed.test.tsruns the service as it runs in production: from the command line, behind a TLS server that does what the load balancer does, with a local GCS server, metadata server, and Sentry server. Each CI job runs the GitHub Action, and its command has only the proxy variables of its run.
The repository is a pnpm workspace with two packages. The root package is
private and has only the tools: oxlint, oxfmt, TypeScript, vitest, and knip.
pnpm-workspace.yaml has the catalog of the versions that the packages
share. The production setup is in deploy/gcp/ and Dockerfile.
packages/roachis the npm package@sentry/roach: the client, the vitest integration, and the action. It has no runtime dependencies, and it ships onlysrc/.tests/package.test.tsfails when an export or the action loads a package that is not a Node built-in or vitest.packages/serviceis the private package@sentry/roach-service: the shared service and the proxy. It imports@sentry/roachas a workspace package (workspace:*). It is the only package that depends on@sentry/node. Its tests run the client, the vitest integration, and the action against a real service.
In packages/roach/src/:
types.ts: rules, modes, runs, sessions, and the results of a run.values.ts: changing values and their placeholders.client.ts: starts a run and calls the control API.config.ts:roach.json, for the action and vitest.vitest.ts: the vitest global setup: one run for each invocation.vitest-setup.ts: the vitest setup file: one session for each test.report.ts: text reports of a run.action.ts: the GitHub Action (action.ymlat the root).
In packages/service/src/:
service.ts: the shared service: tenants, runs, access, and metrics.cli.ts: the command line, which the image runs.server.ts: sockets, HTTPS interception, the allow list, and the control API.bodies.ts: request bodies and their size limits.clients.ts: the PROXY protocol header of the load balancer.run-config.ts: the check of the run config of a client, with its limits.errors.ts: the errors that are the fault of the client, and the log line and the client text of an unexpected error.addresses.ts: the public addresses that the proxy can send requests to.recorder.ts: modes, sessions, replay, and misses.recording-file.ts: a recording file from a live response, the response of a replay, and the decoding of compressed bodies.request-key.ts: the key of a request.misses.ts: request samples and the first change of a miss.store.ts: the formats of recordings and request samples, and the store contract.recordings.ts: the file store, for tests and development.gcs.ts: the GCS store.streams.ts: merges the deltas of a recorded model stream.usage.ts: the token usage of a model response, for Sentry.secrets.ts: redaction of credentials.json-strings.ts: the parse of a JSON body, and the one-pass scan of the JSON strings of a body, forsecrets.ts.certificates.ts: the certificate authority for HTTPS.