Skip to content

Commit 20f4da0

Browse files
docs: Fix the CI, POST /runs, PROXY header, and model attribute notes
CI runs the tests only with coverage. POST /runs also answers 400 for a body that is not JSON and 413 over 1 MiB. Put back the warning about a false PROXY header from step 3 to step 4. roach.written is also on roach.run.end, and a call without a request model has gen_ai.response.model, so the cost query groups by both. Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com>
1 parent 16b9cae commit 20f4da0

2 files changed

Lines changed: 37 additions & 33 deletions

File tree

‎README.md‎

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -900,9 +900,11 @@ once after it.
900900
With `proxyProtocol`, the service reads a PROXY header when a connection
901901
has one, and it uses the address of the connection when it does not. The
902902
load balancer writes its header before the first byte of the client, so a
903-
client cannot put its own header first. A false header changes only the
904-
address for the limit of runs for each client, and the firewall lets only
905-
the Google ranges connect to the VM.
903+
client cannot put its own header first. A client can send its own header
904+
only when the service has `proxyProtocol` and the load balancer does not
905+
send the header: from step 3 to step 4 above. So do step 4 at once. A
906+
false header changes only the address for the limit of runs for each
907+
client, and the firewall lets only the Google ranges connect to the VM.
906908

907909
To add a project, add its repository to the repositories of the
908910
`ROACH_TOKEN` organization secret, in the GitHub settings of the
@@ -943,7 +945,8 @@ The service also has:
943945
- `POST /__roach/runs` with a `RunConfig`: start a run. Send
944946
`Authorization: Bearer <write token>` for any mode but `replay`. Returns
945947
HTTP 201 with the `id`, `token`, proxy `url`, `controlUrl`, and `caCert`
946-
of the run. A config that is not valid gets HTTP 400, a mode other than
948+
of the run. A body that is not JSON or a config that is not valid gets
949+
HTTP 400, a body over 1 MiB gets HTTP 413, a mode other than
947950
`replay` without the token gets HTTP 401, and a run past the limit of
948951
runs without a token gets HTTP 429 (see [Limits](#limits)).
949952
- `DELETE /__roach/runs/<id>`: end the run. Returns its stats.
@@ -990,8 +993,8 @@ pnpm install
990993
pnpm check # format, lint, types, unused code, and tests
991994
```
992995

993-
CI also runs the tests with coverage (`pnpm test:coverage`) and packs
994-
`@sentry/roach`. `AGENTS.md` has the conventions and the commands for one
996+
CI runs the tests only with coverage (`pnpm test:coverage`), and packs
997+
`@sentry/roach`. It does not run `pnpm check`, so run it before you push. `AGENTS.md` has the conventions and the commands for one
995998
file.
996999

9971000
- `packages/roach/tests/package.test.ts` checks what the npm package

‎TELEMETRY.md‎

Lines changed: 28 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -64,32 +64,33 @@ For metrics, use `-f dataset=tracemetrics`,
6464

6565
## Pivots
6666

67-
| Pivot | Meaning | Found In |
68-
| --------------------------- | ------------------------------------------ | --------------------------------- |
69-
| `roach.tenant` | repository of the run | metrics, spans, errors |
70-
| `roach.mode` | `auto`, `replay`, `record`, `off` | metrics, spans, errors |
71-
| `roach.rule` | recording rule, such as `model` | metrics, spans |
72-
| `roach.result` | `replayed`, `live`, and others (see Words) | metrics, spans |
73-
| `roach.origin` | upstream origin | metrics, spans |
74-
| `roach.reason` | why the proxy refused a request | `roach.request` metric and span |
75-
| `roach.run.id` | `name` of the run, else the service id | spans, errors |
76-
| `roach.ci.run_id` | `GITHUB_RUN_ID` of the run | spans, errors |
77-
| `roach.ci.job` | `GITHUB_JOB` of the run | spans, errors |
78-
| `roach.ci.repo` | `GITHUB_REPOSITORY` of the run | spans, errors |
79-
| `roach.ci.ref` | `GITHUB_REF` of the run | spans, errors |
80-
| `roach.session` | session (test) name of a request | `roach.request` |
81-
| `roach.key` | recording key of a request | `roach.key`, `roach.store.read` |
82-
| `roach.write` | the run has the write token (boolean) | `roach.run.start` |
83-
| `roach.recordings` | files that one write sends to the store | `roach.store.write` |
84-
| `roach.written` | files that were new or changed | `roach.store.write` |
85-
| `http.request.method` | method of the request | `roach.request`, `roach.upstream` |
86-
| `http.response.status_code` | status to the client | `roach.request`, `roach.upstream` |
87-
| `gen_ai.request.model` | model of a model call | `roach.request` |
88-
| `miss.sample` | `found`, `withheld`, or `none` for a miss | `roach.request` |
89-
| `miss.changed_at` | offset of the change of a miss | `roach.request` |
90-
| `miss.compared` | recording key of a miss change | `roach.request` |
91-
| `miss.recorded` | recorded text of a miss change | `roach.request` |
92-
| `miss.request` | sent text of a miss change | `roach.request` |
67+
| Pivot | Meaning | Found In |
68+
| --------------------------- | ------------------------------------------------ | ------------------------------------ |
69+
| `roach.tenant` | repository of the run | metrics, spans, errors |
70+
| `roach.mode` | `auto`, `replay`, `record`, `off` | metrics, spans, errors |
71+
| `roach.rule` | recording rule, such as `model` | metrics, spans |
72+
| `roach.result` | `replayed`, `live`, and others (see Words) | metrics, spans |
73+
| `roach.origin` | upstream origin | metrics, spans |
74+
| `roach.reason` | why the proxy refused a request | `roach.request` metric and span |
75+
| `roach.run.id` | `name` of the run, else the service id | spans, errors |
76+
| `roach.ci.run_id` | `GITHUB_RUN_ID` of the run | spans, errors |
77+
| `roach.ci.job` | `GITHUB_JOB` of the run | spans, errors |
78+
| `roach.ci.repo` | `GITHUB_REPOSITORY` of the run | spans, errors |
79+
| `roach.ci.ref` | `GITHUB_REF` of the run | spans, errors |
80+
| `roach.session` | session (test) name of a request | `roach.request` |
81+
| `roach.key` | recording key of a request | `roach.key`, `roach.store.read` |
82+
| `roach.write` | the run has the write token (boolean) | `roach.run.start` |
83+
| `roach.recordings` | files that one write sends to the store | `roach.store.write` |
84+
| `roach.written` | files that were new or changed | `roach.store.write`, `roach.run.end` |
85+
| `http.request.method` | method of the request | `roach.request`, `roach.upstream` |
86+
| `http.response.status_code` | status to the client | `roach.request`, `roach.upstream` |
87+
| `gen_ai.request.model` | model that a model call asked for | `roach.request` |
88+
| `gen_ai.response.model` | model of the response, when the request has none | `roach.request` |
89+
| `miss.sample` | `found`, `withheld`, or `none` for a miss | `roach.request` |
90+
| `miss.changed_at` | offset of the change of a miss | `roach.request` |
91+
| `miss.compared` | recording key of a miss change | `roach.request` |
92+
| `miss.recorded` | recorded text of a miss change | `roach.request` |
93+
| `miss.request` | sent text of a miss change | `roach.request` |
9394

9495
A metric for a `notAllowed` request has `roach.origin:other`, because the
9596
client picks that origin. Its span has the origin.
@@ -255,7 +256,7 @@ Spent and saved, by rule and model.
255256
```bash
256257
sentry explore sentry/roach --dataset spans --period 24h \
257258
-q 'span.op:roach.request has:gen_ai.usage.input_tokens' \
258-
-F roach.result -F roach.rule -F gen_ai.request.model \
259+
-F roach.result -F roach.rule -F gen_ai.request.model -F gen_ai.response.model \
259260
-F 'sum(gen_ai.cost.total_tokens)' -F 'sum(gen_ai.usage.input_tokens)' \
260261
-F 'sum(gen_ai.usage.output_tokens)' -F 'count()'
261262
```

0 commit comments

Comments
 (0)