Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
150 changes: 106 additions & 44 deletions README.md

Large diffs are not rendered by default.

19 changes: 17 additions & 2 deletions TELEMETRY.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,11 @@ For metrics, use `-f dataset=tracemetrics`,
`roach.reason` tells why: `notAllowed` (the origin is not in `allow`),
`notPublic` (the address is not public), or `replayOnly` (a run without
the token, and no rule records the request).
- **Busy**: the body of a run without the token did not fit in the
budget of bodies, and the proxy answered HTTP 503 with `Retry-After`.
The run goes on, and the client can send the request again.
- **Timeout**: the body of a run without the token did not come in 30 s,
and the proxy answered HTTP 408.
- **Dropped**: a live response was recorded, but its session failed, so the
recording was not written. The next run sends the request live again.

Expand Down Expand Up @@ -96,8 +101,8 @@ The examples use the `sentry` CLI. The same fields work in Explore. The
counts are estimates (see [Sampling](#sampling)).

Cached and uncached requests by rule. The hit rate of a rule is
`replayed / (replayed + live + missed)`. Do not count `passthrough` or
`refused`.
`replayed / (replayed + live + missed)`. Do not count `passthrough`,
`refused`, `busy`, or `timeout`.

```bash
sentry explore sentry/roach --dataset metrics --metric roach.request --agg sum \
Expand All @@ -121,6 +126,16 @@ sentry explore sentry/roach --dataset metrics --metric roach.request --agg sum \
-F roach.origin
```

Requests of runs without the token that the budget of bodies stopped
(`busy`, HTTP 503) or that were too slow (`timeout`, HTTP 408), by tenant.
A steady count of `busy` from many tenants shows that the budget is too
small. A count from one tenant shows one client that fills its part.

```bash
sentry explore sentry/roach --dataset metrics --metric roach.request --agg sum \
--period 24h -q 'roach.result:[busy,timeout]' -F roach.tenant -F roach.result
```

Time saved by a replay. Compare `live` and `replayed` for each rule. Use
`--agg p95` for the slow tail.

Expand Down
68 changes: 47 additions & 21 deletions packages/roach/src/values.ts
Original file line number Diff line number Diff line change
Expand Up @@ -155,7 +155,8 @@ export type KnownValues = ReadonlyMap<string, string>;
* value at the same place.
*
* Past `maxValues` values, it stops and throws an error with HTTP status
* 400. So a text with millions of values cannot fill the memory.
* 400. It counts each value when it finds it, so a text with millions of
* values cannot fill the memory before the error.
*/
export function extractValues(
text: string,
Expand All @@ -174,28 +175,53 @@ export function extractValues(
const search = hasKnown
? new RegExp(`${pattern.source}|(${literals([...known.keys()])})`, "g")
: pattern;
// An `exec()` loop, not `replace()`: `replace()` finds all matches
// before it calls its function, so it cannot stop at `maxValues`. The
// loop finds the same matches, also the empty ones.
const parts: string[] = [];
let count = 0;
const result = text.replace(search, (...match: unknown[]) => {
count += 1;
if (count > maxValues) {
throw Object.assign(
new Error("the request has too many changing values"),
{ status: 400 },
);
let end = 0;
search.lastIndex = 0;
try {
for (let match = search.exec(text); match; match = search.exec(text)) {
count += 1;
if (count > maxValues) {
throw Object.assign(
new Error("the request has too many changing values"),
{ status: 400 },
);
}
const value = match[0];
const name =
hasKnown && match[knownGroup] !== undefined
? known.get(value)!
: names[groups.findIndex((group) => match[group] !== undefined)]!;
const list = values.get(name) ?? [];
list.push(value);
values.set(name, list);
parts.push(text.slice(end, match.index), `<<${name}>>`);
end = match.index + value.length;
if (value.length === 0) search.lastIndex += 1;
}
const value = match[0] as string;
const name =
hasKnown && match[knownGroup] !== undefined
? known.get(value)!
: names[groups.findIndex((group) => match[group] !== undefined)]!;
const list = values.get(name) ?? [];
list.push(value);
values.set(name, list);
return `<<${name}>>`;
});
return { text: result, values };
} finally {
// `pattern` is shared, so it must not keep a position.
search.lastIndex = 0;
}
parts.push(text.slice(end));
return { text: parts.join(""), values };
}

/**
* Whether `value` can be in `known` or in a template (`templateValues()`).
* V8 cannot run a pattern with a literal of over 32,767 characters on
* any text, also on a text of 5 characters. (A literal with a character
* over U+00FF fails only on a text with such a character, and only such a
* text can hold it.) Such a value in `known` failed each later request of
* its session, so those requests had no key, and to leave it out changes
* no key.
*/
export const canBeKnown = (value: string): boolean => value.length <= 32_767;

const escape = (text: string) => text.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");

/**
Expand All @@ -212,13 +238,13 @@ const literals = (values: string[]) =>
/**
* Write each value of the request that a recorded response repeats as
* `<<name:n>>`. A value is replaced only where it is not part of a longer
* word or number.
* word or number. A value that `canBeKnown()` refuses stays as it is.
*/
export function templateValues(text: string, values: RequestValues): string {
const placeholders = new Map<string, string>();
for (const [name, list] of values) {
list.forEach((value, index) => {
if (!placeholders.has(value)) {
if (!placeholders.has(value) && canBeKnown(value)) {
placeholders.set(value, `<<${name}:${index + 1}>>`);
}
});
Expand Down
42 changes: 22 additions & 20 deletions packages/service/src/key-worker.ts
Original file line number Diff line number Diff line change
@@ -1,13 +1,15 @@
/**
* The worker process that makes keys and redacts the text of misses.
* The worker process that makes the keys of runs without the token, and
* redacts the text of their misses.
*
* The value patterns of a run come from its client, and redaction takes
* time for each credential that a run knows. Both run in this worker, so a
* slow request does not stop the event loop that answers all runs. The
* service has one worker. It does one job at a time, and each job has a
* time limit (the `timeout` of `vm`). The jobs of runs with the write token
* go before the jobs of runs without it. A job of an ended run does not
* start.
* Anyone can start a run without the token, and its value patterns and
* bodies come from its client. Redaction takes time for each credential
* that a run knows. Both run in this worker, so a slow request does not
* stop the event loop that answers all runs. The service has one worker.
* It does one job at a time, in order, and each job has a time limit (the
* `timeout` of `vm`). A job of an ended run does not start. A run with the
* token does this work on the main thread (`recorder.ts`), because the
* key of a 64 MiB body can need more memory than the worker has.
*
* The worker is a child process with a heap limit (`MAX_HEAP_MB`), so one
* request cannot fill the memory of the service. It is not a worker
Expand Down Expand Up @@ -68,8 +70,6 @@ export interface JobOptions {
timeLimitMs: number;
/** The most changing values that a key can have. */
maxValues: number;
/** Run the job before the jobs that do not have this. */
first: boolean;
/** Called before the job starts. When it returns `true`, it does not. */
skip(): boolean;
}
Expand All @@ -90,11 +90,13 @@ interface JobResult<T> {

const MARK = "roach-key-worker";
/**
* The old-generation heap of the worker, in MiB. The key of a 64 MiB body,
* the largest body of a run with the token, needs about 320 MiB. At 256 MiB,
* a 64 MiB body with a screenshot fails.
* The heap of the worker, in MiB: the old generation and each semi-space
* of the young generation. The worker makes the keys of runs without the
* token, with bodies of at most 16 MiB. The key of a 16 MiB body of small
* text blocks peaks at about 330 MiB of RSS. The worst body, such as 16 MiB
* of `{}`, stops the worker at about 500 MiB of RSS.
*/
const MAX_HEAP_MB = 384;
const MAX_HEAP_MB = { old: 256, semi: 4 };
/** The time after the time limit of a job before the worker stops. */
const WATCHDOG_MARGIN_MS = 1000;

Expand Down Expand Up @@ -163,7 +165,11 @@ function failWorker(stopped: ChildProcess, why: string): void {

function startWorker(): ChildProcess {
const started = fork(fileURLToPath(import.meta.url), [MARK], {
execArgv: [...process.execArgv, `--max-old-space-size=${MAX_HEAP_MB}`],
execArgv: [
...process.execArgv,
`--max-old-space-size=${MAX_HEAP_MB.old}`,
`--max-semi-space-size=${MAX_HEAP_MB.semi}`,
],
// `Map` values, such as `known`, need the advanced serialization.
serialization: "advanced",
stdio: ["ignore", "inherit", "inherit", "ipc"],
Expand Down Expand Up @@ -205,11 +211,7 @@ function hold(child: ChildProcess | undefined, busy: boolean): void {
/** Start the next job, when the worker has none. */
function next(): void {
while (!running && queue.length > 0) {
const index = Math.max(
0,
queue.findIndex((pending) => pending.first),
);
const [pending] = queue.splice(index, 1) as [Pending];
const pending = queue.shift()!;
if (pending.skip()) {
pending.reject(jobError("this run has ended", { status: 410 }));
continue;
Expand Down
Loading
Loading