Skip to content

Commit 09f6582

Browse files
authored
fix(secrets): Check escapes, fail closed on hidden fields, learn \/ (#47)
2 parents 9b2ca6a + 59764ec commit 09f6582

8 files changed

Lines changed: 1112 additions & 126 deletions

File tree

‎README.md‎

Lines changed: 31 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -231,7 +231,10 @@ A run takes `RunConfig` (`packages/roach/src/types.ts`): `tenant`, `mode`, `rule
231231
1 Mi characters. A run with the token learns at most 16,384 values, with
232232
a total of 16 Mi characters, for credentials that change on each
233233
request, such as AWS signatures. The whole value and each part of it
234-
count, also for the `secrets` of the run. A run whose `secrets` go past
234+
count, also for the `secrets` of the run. The proxy also redacts each
235+
part with `\/` for `/`, as some servers write it in JSON, and the whole
236+
value when a `/` is only in parts shorter than 12 characters. These
237+
forms do not count. A run whose `secrets` go past
235238
the limit gets HTTP 400 when it starts. Past the limit, a request with a
236239
new credential gets HTTP 431, and the proxy does not send or record it.
237240
A request with known credentials goes on. Start a new run to learn more.
@@ -305,7 +308,24 @@ A run takes `RunConfig` (`packages/roach/src/types.ts`): `tenant`, `mode`, `rule
305308
`secret_key`, `private_key`, and `secret_access_key`. In JSON, these
306309
fields count at each depth. In an `application/x-www-form-urlencoded`
307310
body, the form fields count. No key uses this text, so this list can be
308-
longer.
311+
longer. The proxy replaces the value at the place of each field, and
312+
each other string that is the same value. A value pattern can change
313+
the text of a key, so then the proxy checks the result. It parses the
314+
body again, and the body must be JSON. Each field must hold
315+
`<<redacted>>` at its place, and no string can be one of the values.
316+
Only the body counts, not a key header. When a check fails, the request
317+
sample has no text, and the proxy writes a log line with the key. This
318+
occurs when a value pattern takes a quote, or a name of a field, away.
319+
A request with credential fields also gets no text when the name of a
320+
value of its rule has `"`, `\`, `&`, `=`, `<`, `>`, or a control
321+
character, because its `<<name>>` can add a field. A miss has no change
322+
then.
323+
- The proxy cannot find the fields of a body that is not JSON or a form,
324+
such as JSON with a comma at the end, JSON lines, or a multipart body.
325+
So the request sample of such a body has no text when the body has the
326+
name of a field above anywhere. The name can be in any case, with `_`,
327+
`-`, or nothing between its words, such as `accessToken`. This is one
328+
search of the body, so a long body costs little.
309329
- The proxy parses a JSON body to find the fields, so a long string value
310330
cannot stop it. It does not find a field whose value is not a string,
311331
such as an array, or a field in JSON text that is itself a string, such
@@ -462,8 +482,10 @@ shows it.
462482
change would let anyone guess the requests of a tenant back, one
463483
character at a time.
464484
- A miss has no change when there is no sample, such as for a recording
465-
from an older version, or when the texts are the same in the part that
466-
the sample keeps.
485+
from an older version, when the sample has no text (`withheld`, see
486+
[Config](#config)), or when the texts are the same in the part that the
487+
sample keeps. A sample with no text still takes the place of the older
488+
sample, so a miss never compares with a request that is not the last.
467489
- The proxy redacts the text of the miss before it compares it. This has
468490
the time limit of a key, and for a run without the token it runs in the
469491
worker process of the keys (see [Changing values](#changing-values)).
@@ -626,10 +648,11 @@ Each proxied request is one trace. Each span has `roach.tenant`,
626648
response has token usage, the span also has the `gen_ai` attributes of
627649
the model call (`usage.ts`). Sentry then calculates its cost in USD as
628650
`gen_ai.cost.total_tokens`. The span of the first miss of a session also
629-
has `miss.sample` (`found` or `none`), in every run. When the proxy finds
630-
a change, the span also has `miss.changed_at` (the offset of the first
631-
different character), `miss.compared` (the key of the recording), and
632-
the redacted text of the change: `miss.recorded` and `miss.request`.
651+
has `miss.sample` (`found`, `withheld`, or `none`), in every run. When
652+
the proxy finds a change, the span also has `miss.changed_at` (the
653+
offset of the first different character), `miss.compared` (the key of
654+
the recording), and the redacted text of the change: `miss.recorded` and
655+
`miss.request`.
633656
Only Sentry gets these texts ([Misses](#misses)). It contains:
634657
- `roach.key`: the key of the request, with `roach.rule` and
635658
`roach.key`.

‎TELEMETRY.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -81,7 +81,7 @@ For metrics, use `-f dataset=tracemetrics`,
8181
| `roach.write` | the run has the write token (boolean) | `roach.run.start` |
8282
| `http.response.status_code` | status to the client | `roach.request`, `roach.upstream` |
8383
| `gen_ai.request.model` | model of a model call | `roach.request` |
84-
| `miss.sample` | `found` or `none` for a miss | `roach.request` |
84+
| `miss.sample` | `found`, `withheld`, or `none` for a miss | `roach.request` |
8585
| `miss.changed_at` | offset of the change of a miss | `roach.request` |
8686
| `miss.compared` | recording key of a miss change | `roach.request` |
8787
| `miss.recorded` | recorded text of a miss change | `roach.request` |
@@ -185,7 +185,9 @@ sentry explore sentry/roach --dataset spans --period 24h \
185185
Where the requests of misses changed, and what changed. The same
186186
`miss.changed_at` from run to run shows one value that changes on each
187187
run. `miss.recorded` and `miss.request` show that value. `miss.sample:none`
188-
is a miss that has no recorded request to compare with. This works for
188+
is a miss that has no recorded request to compare with.
189+
`miss.sample:withheld` is a miss whose recorded request had a credential
190+
that the proxy could not redact, so its sample has no text. This works for
189191
runs without the token too, such as CI of a fork. Only Sentry has these
190192
texts: the run report shows only the offset and the key, because CI logs
191193
can be public.

‎packages/service/src/key-worker.ts‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -279,14 +279,15 @@ export function keyInWorker(
279279

280280
/**
281281
* `redactRequestWith()` in the worker, with the time limit of `options`.
282-
* Past it, the job fails with `timedOut`.
282+
* Past it, the job fails with `timedOut`. The value is `undefined` when a
283+
* credential field cannot be redacted.
283284
*/
284285
export function redactInWorker(
285286
values: string[],
286287
text: string,
287288
request: RequestToRedact,
288289
options: JobOptions,
289-
): Promise<JobResult<string>> {
290+
): Promise<JobResult<string | undefined>> {
290291
return runInWorker(
291292
{ kind: "redact", values, text, request, timeLimitMs: options.timeLimitMs },
292293
options,

‎packages/service/src/misses.ts‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,15 @@ export function toSample(file: string, text: string): RequestSample {
5757
: { file, text };
5858
}
5959

60+
/**
61+
* The sample of a recorded request whose text the proxy could not redact.
62+
* It has no text, but it takes the place of an older sample, so a later
63+
* miss does not compare with a request that is not the last one.
64+
*/
65+
export function withheldSample(file: string): RequestSample {
66+
return { file, text: "", withheld: true };
67+
}
68+
6069
/**
6170
* The first change from the text of `sample` to `text`. `undefined` when
6271
* the texts are the same, or when they differ only after the part that

‎packages/service/src/recorder.ts‎

Lines changed: 26 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -35,15 +35,20 @@ import type { IncomingHttpHeaders } from "node:http";
3535
import { promisify } from "node:util";
3636
import { brotliDecompress, gunzip, inflate, zstdDecompress } from "node:zlib";
3737
import { reportInternal } from "./errors.ts";
38-
import { firstChange, sampleKey, toSample } from "./misses.ts";
38+
import { firstChange, sampleKey, toSample, withheldSample } from "./misses.ts";
3939
import {
4040
keyInWorker,
4141
redactInWorker,
4242
type JobError,
4343
type JobOptions,
4444
} from "./key-worker.ts";
4545
import { keyRequest, keyTimeLimit, type KeyedRequest } from "./request-key.ts";
46-
import { createSecrets, redactUrl, type LearnLimit } from "./secrets.ts";
46+
import {
47+
createSecrets,
48+
redactUrl,
49+
toRedact,
50+
type LearnLimit,
51+
} from "./secrets.ts";
4752
import {
4853
recordingKey,
4954
RULE_NAME,
@@ -571,13 +576,16 @@ export function createRecorder(config: RecorderConfig, store: RecordingStore) {
571576
const places = owner ?? outside;
572577
if (places.explained) return;
573578
places.explained = true;
574-
const recorded = (await store.read(sample).catch(() => undefined)) as
579+
const read = (await store.read(sample).catch(() => undefined)) as
575580
| RequestSample
576581
| undefined;
582+
const recorded = read?.withheld ? undefined : read;
577583
// The redacted text has a time limit. Past it, the change is left out.
578584
const redacted = recorded && (await text());
579585
const change = redacted && firstChange(recorded, redacted);
580-
telemetry.annotate({ "miss.sample": recorded ? "found" : "none" });
586+
telemetry.annotate({
587+
"miss.sample": read ? (recorded ? "found" : "withheld") : "none",
588+
});
581589
if (!change) return;
582590
telemetry.annotate({
583591
"miss.changed_at": change.at,
@@ -722,7 +730,7 @@ export function createRecorder(config: RecorderConfig, store: RecordingStore) {
722730
if (!workLimit.untrusted) {
723731
try {
724732
return withTimeLimit(keyTimeLimit(keyed.text.length), () =>
725-
secrets.redactRequest(keyed.text, request),
733+
secrets.redactRequest(keyed.text, request, rule),
726734
);
727735
} catch {
728736
return undefined;
@@ -731,11 +739,7 @@ export function createRecorder(config: RecorderConfig, store: RecordingStore) {
731739
return redactInWorker(
732740
secrets.list(),
733741
keyed.text,
734-
{
735-
href: request.url.href,
736-
body: request.body.toString("utf8"),
737-
contentType: request.headers["content-type"],
738-
},
742+
toRedact(request, rule),
739743
jobOptions(keyed.text.length),
740744
).then(
741745
(result) => {
@@ -855,10 +859,21 @@ export function createRecorder(config: RecorderConfig, store: RecordingStore) {
855859
} catch (error) {
856860
return notRecorded("it could not be redacted", error);
857861
}
862+
const text = secrets.redactRequest(keyed.text, request, rule);
863+
if (text === undefined) {
864+
// A credential field of the request can stay in the text, so the
865+
// recorder writes a sample without text.
866+
process.stderr.write(
867+
`[roach] The ${rule.name} request sample has no text, because a credential field could not be redacted: ${key}\n`,
868+
);
869+
}
858870
await record(owner, key, {
859871
recording: stored,
860872
samples: [
861-
[sample, toSample(key, secrets.redactRequest(keyed.text, request))],
873+
[
874+
sample,
875+
text === undefined ? withheldSample(key) : toSample(key, text),
876+
],
862877
],
863878
});
864879
}

0 commit comments

Comments
 (0)