Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 11 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -606,11 +606,11 @@ attribute of Roach starts with `roach.`. Metrics have only attributes with
few values: `roach.tenant`, `roach.mode`, `roach.rule`, `roach.origin`,
`roach.result`, and `roach.reason`.

| Metric | Type | Value |
| ------------------------ | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `roach.request` | counter | 1 for each proxied request. `roach.result` is `replayed`, `live`, `missed`, `passthrough`, `refused`, `busy` (HTTP 503: the body did not fit in the budget), or `timeout` (HTTP 408: the body did not come in time). A `passthrough`, `refused`, `busy`, or `timeout` request has no `roach.rule`. A `refused` request has `roach.reason`, and `other` for an origin not in `allow`. |
| `roach.request.duration` | distribution (ms) | The time to answer a request that a rule records, by `roach.result`. |
| `roach.recording.write` | counter | Recordings by `roach.rule`. `roach.result` is `written` (the proxy sent them to the store) or `dropped` (their session failed). |
| Metric | Type | Value |
| ------------------------ | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `roach.request` | counter | 1 for each proxied request. `roach.result` is `replayed`, `live`, `missed`, `passthrough`, `refused`, `busy` (HTTP 503: the body did not fit in the budget), `timeout` (HTTP 408: the body did not come in time), or `aborted` (the client closed the request before its body came). A `passthrough`, `refused`, `busy`, `timeout`, or `aborted` request has no `roach.rule`. A `refused` request has `roach.reason`, and `other` for an origin not in `allow`. |
| `roach.request.duration` | distribution (ms) | The time to answer a request that a rule records, by `roach.result`. |
| `roach.recording.write` | counter | Recordings by `roach.rule`. `roach.result` is `written` (the proxy sent them to the store) or `dropped` (their session failed). |

Each proxied request is one trace. Each span has `roach.tenant`,
`roach.mode`, `roach.run.id` (the `name` of the run, else its id), and the
Expand Down Expand Up @@ -698,7 +698,7 @@ To see how well the cache works:
run without the token. A control API body over 1 MiB gets HTTP 413
before the service parses it.
- The requests of all runs without the token can hold 128 MiB of bodies
at one time, and the runs of one client 32 MiB. The client of a run is
at one time, and the runs of one client 16 MiB. The client of a run is
the client that started it, grouped as for the limit of runs. A request
holds its body until it finishes, also while its key waits for the key
worker. A request whose body does not fit gets HTTP 503 with
Expand All @@ -714,6 +714,9 @@ To see how well the cache works:
2.0 GB.
- Keys and the redaction of misses have time limits. See
[Changing values](#changing-values).
- Request headers that do not come in 10 s get HTTP 408, and a request
whose body does not come in 300 s too. This is also true in a tunnel
and with `proxyProtocol`.
- A `CONNECT` tunnel that sends no byte for 10 s after the `200` closes. A
tunnel that sends and gets no byte for 10 minutes closes.
- The credentials that a run learns and the values that the sessions of
Expand All @@ -722,13 +725,8 @@ To see how well the cache works:
- A gap: a client with an IPv6 /48 has 65,536 /64s, so it can open runs
until the service has 100 runs without a token. Then other clients get
HTTP 429 until its runs end. Its runs can also hold all of the budget of
bodies, for 30 s at a time.
- A gap: Node times out the headers and the body of a request only on a
server that listens. The servers of tunnels, and the HTTP server with
`proxyProtocol`, get their connections from another server. So a
request of a run with the token, and a control API call with
`proxyProtocol`, have no time limit for their headers and body. A
tunnel closes only after 10 minutes without a byte.
bodies. A body gets HTTP 408 after 30 s, but the client can start the
next one at once, so it holds the budget for as long as it sends.
- GCS deletes each recording 30 days after it was written, also when runs
still replay it. Then `auto` mode records it again, and `replay` mode
fails the request with HTTP 412.
Expand Down
5 changes: 4 additions & 1 deletion TELEMETRY.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,9 @@ For metrics, use `-f dataset=tracemetrics`,
The run goes on, and the client can send the request again.
- **Timeout**: the body of a run without the token did not come in 30 s,
and the proxy answered HTTP 408.
- **Aborted**: the client closed its connection before the body of its
request came. The proxy sent the request nowhere. It is not an error,
so Sentry gets no event.
- **Dropped**: a live response was recorded, but its session failed, so the
recording was not written. The next run sends the request live again.

Expand Down Expand Up @@ -102,7 +105,7 @@ counts are estimates (see [Sampling](#sampling)).

Cached and uncached requests by rule. The hit rate of a rule is
`replayed / (replayed + live + missed)`. Do not count `passthrough`,
`refused`, `busy`, or `timeout`.
`refused`, `busy`, `timeout`, or `aborted`.

```bash
sentry explore sentry/roach --dataset metrics --metric roach.request --agg sum \
Expand Down
55 changes: 44 additions & 11 deletions packages/service/src/server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,15 @@ const TUNNEL_FIRST_BYTE_MS = 10_000;
* call can take minutes before its first byte, so this is long.
*/
const TUNNEL_IDLE_MS = 10 * 60 * 1000;
/**
* The HTTP servers of the proxy. Headers that do not come in 10 s get
* HTTP 408 from Node. A whole request, with its body, has Node's default
* of 300 s. Node checks these times each second.
*/
const HTTP_OPTIONS: http.ServerOptions = {
headersTimeout: 10_000,
connectionsCheckingInterval: 1000,
};
/**
* A PROXY protocol v1 header, such as
* `PROXY TCP4 203.0.113.7 10.0.0.2 51234 443`. It is at most 107 bytes.
Expand Down Expand Up @@ -248,9 +257,8 @@ function readBody(
clearTimeout(deadline);
reject(error);
};
// Node does not time out a request in a tunnel (`listenProxy()`), so
// a client that sends a byte now and then could hold its bytes for
// as long as the tunnel stays open.
// Node gives a whole request 300 s (`HTTP_OPTIONS`). That is too long
// to hold a part of the budget, so these bodies have a shorter limit.
const readMs = held.budget?.readMs;
const deadline =
readMs === undefined
Expand All @@ -277,13 +285,24 @@ function readBody(
clearTimeout(deadline);
resolve(Buffer.concat(chunks));
});
stream.on("error", (error) => {
stream.on("error", (error: Error & { code?: string }) => {
clearTimeout(deadline);
reject(error);
// A client that closes its connection before its body came is not a
// failure of the proxy, so it does not go to Sentry. `method` is set
// only on a request, not on an upstream response.
const aborted = stream.method !== null && error.code === "ECONNRESET";
reject(aborted ? abortedError() : error);
});
});
}

/** The error of a request whose client closed it before its body came. */
const abortedError = () =>
Object.assign(
httpError(400, "the client closed the request before its body came"),
{ aborted: true },
);

/**
* Read a JSON object body of a control API call. A body over
* `MAX_CONTROL_BODY_BYTES` fails with HTTP 413 before it is parsed. Any
Expand Down Expand Up @@ -712,10 +731,16 @@ function answerError(
status === undefined
? reportInternal("Request", error, target.telemetry.capture(error))
: (error as Error).message;
// A body that did not fit in the budget, or that did not come in time,
// is not a miss and not an error. Count it, so that Sentry shows it.
const result =
status === 503 ? "busy" : status === 408 ? "timeout" : undefined;
// A body that did not fit in the budget, that did not come in time, or
// whose client closed the request is not a miss and not an error. Count
// it, so that Sentry shows it.
const result = (error as { aborted?: boolean }).aborted
? "aborted"
: status === 503
? "busy"
: status === 408
? "timeout"
: undefined;
if (result) {
target.telemetry.annotate({ "roach.result": result });
target.telemetry.count("roach.request", 1, {
Expand Down Expand Up @@ -754,15 +779,15 @@ export async function listenProxy(
const sockets = new Set<Socket>();

// Requests inside a CONNECT tunnel. The tunnel gives the origin.
const tunnelServer = http.createServer((incoming, outgoing) => {
const tunnelServer = http.createServer(HTTP_OPTIONS, (incoming, outgoing) => {
const tunnel = tunnels.get(incoming.socket);
if (!tunnel) {
outgoing.writeHead(400).end();
return;
}
serveProxied(tunnel, incoming, outgoing, tunnel.origin);
});
const server = http.createServer((incoming, outgoing) => {
const server = http.createServer(HTTP_OPTIONS, (incoming, outgoing) => {
// A request to the proxy itself has a path. A proxied plain HTTP
// request has an absolute URL.
if (!incoming.url?.startsWith("/")) {
Expand Down Expand Up @@ -878,6 +903,14 @@ export async function listenProxy(
},
);

// Node checks `headersTimeout` and `requestTimeout` only after the
// `listening` event of a server. The tunnel server, and `server` with
// `proxyProtocol`, get their connections from another server and never
// listen. So send the event to them. This relies on Node internals
// (`setupConnectionsTracking` in `lib/_http_server.js`), and the test
// "times out slow headers" fails when Node changes them.
tunnelServer.emit("listening");
if (listener !== server) server.emit("listening");
await new Promise<void>((resolve, reject) => {
listener.once("error", reject);
listener.listen(options.port, options.host, () => resolve());
Expand Down
6 changes: 3 additions & 3 deletions packages/service/src/service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -173,10 +173,10 @@ const PUBLIC_BODY_BUDGET_BYTES = 128 * 1024 * 1024;
/**
* The part of `PUBLIC_BODY_BUDGET_BYTES` that the runs of one client
* (`clientGroup()` of the client that started the run) can hold, so that
* one client cannot take all of it. It is 2 bodies of 16 MiB, or about 50
* of the largest model requests.
* one client cannot take all of it. It takes 8 clients to fill the budget.
* It is 1 body of 16 MiB, or about 25 of the largest model requests.
*/
const PUBLIC_BODY_CLIENT_BYTES = 32 * 1024 * 1024;
const PUBLIC_BODY_CLIENT_BYTES = 16 * 1024 * 1024;
/**
* A body of a run without a token that does not come in this time gets
* HTTP 408, and gives its bytes back. So a client that sends a byte now
Expand Down
Loading
Loading