Skip to content

fix(server): Enforce request timeouts on servers that do not listen - #49

Merged
dcramer merged 3 commits into
mainfrom
fix/listen-timeouts
Nov 5, 2026
Merged

dcramer merged 3 commits into
mainfrom
fix/listen-timeouts

Conversation

@sentry-junior

@sentry-junior sentry-junior Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Node checks headersTimeout and requestTimeout only on a server that emitted 'listening' (setupConnectionsTracking in lib/_http_server.js). The tunnel server never calls listen(). With proxyProtocol on, the main server does not call it either, and deploy/gcp/main.tf already sets proxyProtocol = true. So a client that sends headers slowly can keep a connection open with no time limit. This is true for runs with and without the write token.

Changes

  • listenProxy() calls tunnelServer.emit("listening"), and server.emit("listening") when proxyProtocol is on. A comment says that this depends on Node internals.
  • Both HTTP servers set headersTimeout: 10_000 (Node's default is 60 s) and connectionsCheckingInterval: 1000. 10 s is the same as the existing PROXY-header and tunnel first-byte limits. requestTimeout stays at Node's default of 300 s.
  • When a client resets its request body (ECONNRESET on the request, not on the upstream response), the error gets status 400. answerError does not send it to Sentry or log "Request failed". It counts as roach.request with roach.result:aborted. Before this change, any client that could start a run could create Sentry error events cheaply.
  • README: metric row and Limits section (10 s headers, 300 s request). TELEMETRY.md: aborted definition and hit-rate exclusion.

Tests

  • "times out slow headers, also with proxyProtocol and in a tunnel": on a proxyProtocol: true service it sends unfinished headers on the main server (after a PROXY header) and inside a tunnel, and expects 408 on both. On main, both connections are still open at 15 s. It also fails if only the two emit lines are removed.
  • "does not send a request to Sentry when its client closes it": on main it fails with [roach] Request failed: Error ECONNRESET. With a local Sentry sink, this branch sends 0 error events and 1 aborted metric with the tenant.
  • pnpm check and pnpm test:coverage pass on Node 24.22.2 (71 tests). No key code changed.

Checked by hand with the new 10 s limit: a body sent over 16 s after fast headers still gets its normal answer, because the headers limit stops once headers are complete.

Merge notes

via @dcramer.

--

View Session [Sentry]

…d do not report aborted requests

Node checks `headersTimeout` and `requestTimeout` only after the
`listening` event. The tunnel server never listens, and with
`proxyProtocol` the HTTP server does not either, so their requests had
no time limit. Send the event to them, and give headers 10 s. Node checks
each second.

A client that closes its connection before its body came gave an
unexpected ECONNRESET, so it went to Sentry. It now gets status 400 and
counts as `roach.request` with `roach.result:aborted`.

Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com>
Keep both sides of #48 and of this branch:
- server.ts: `readBody` clears the 30 s read deadline on end and on an
  error, and an ECONNRESET of a request still becomes the 400 `aborted`
  error. `answerError` counts `aborted`, `busy` (503), and `timeout`
  (408) as `roach.request`. The comment of the read deadline now says
  why it is shorter than Node's 300 s.
- service.test.ts: keep both sets of helpers. `startBody` uses
  `openTunnel`. The aborted test also checks that a closed request gives
  its bytes of the budget back.
- README.md: one metric row with all results. Remove the gap about Node
  time limits, because this branch fixes it.
- TELEMETRY.md: Busy, Timeout, and Aborted, and leave all three out of
  the hit rate.
With 32 MiB, 4 IPv4 addresses or 4 /64s could fill the 128 MiB budget
of runs without a token. With 16 MiB it takes 8. 16 MiB is still one
body of the largest size, or about 25 of the largest model requests.

The README said that a client with a /48 holds the budget "for 30 s at
a time". A body gets HTTP 408 after 30 s, but the client can start the
next one at once, so the hold does not stop.
@dcramer
dcramer marked this pull request as ready for review November 5, 2026 16:04
@dcramer
dcramer merged commit 9b2ca6a into main Nov 5, 2026
16 checks passed
@dcramer
dcramer deleted the fix/listen-timeouts branch November 5, 2026 16:04
dcramer added a commit that referenced this pull request Nov 5, 2026
CI already runs on push to `main` (`ci.yml` has `push: branches: [main,
"release/**"]`). But GitHub started no push workflows for `09f6582` (the
#47 merge): no CI, no Image, no CodeQL. The Deploy run of `9b2ca6a`
(#49) then saw a newer head and stopped ("main is at 09f6582 now, not
9b2ca6a. Nothing deploys."). So the head of `main` has no CI, no image
and no deploy, and `:main` still points at `6c64d29` (#48). Nothing
could start those runs again: neither workflow had `workflow_dispatch`.

**Changes**
- `ci.yml` and `image.yml`: add `workflow_dispatch`.
- `image.yml`: push `:<sha>` on every event except `pull_request`, so a
manual run on `main` makes the image.
- `deploy.yml`:
- `promote` accepts an Image run that isn't a `pull_request` (push or
manual). The `branches: [main]` filter still limits Deploy to `main`.
- "Wait for CI" moves to `.github/scripts/wait-for-ci.sh` and looks for
the CI run of the commit with `--branch main`, not `--event push`. A
manual CI run counts. A `pull_request` run of the same commit doesn't,
because its branch isn't `main`.
- `README.md`: "Deploy" says what to run when a push gets no workflows.

**After merge:** this merge is a new push, so it should get CI, Image
and Deploy normally. If it doesn't, run `gh workflow run ci.yml --ref
main` and `gh workflow run image.yml --ref main`.

**Checked**
- shellcheck (`--shell=sh --enable=check-extra-masked-returns`) and
actionlint pass. `oxfmt --check` passes.
- The script ran under busybox `ash` with stub `gh`, `timeout` and
`sleep`: success after two empty lists, a failed run, a run that never
ends (60 polls, then exit 1), a `gh` error, and no arguments (exit 2).
- The live query `gh run list --workflow ci.yml --branch main --commit
<sha>` finds the push run for `9b2ca6a`, nothing for `09f6582`, and
nothing for `3f11cb0` (#49's PR head, which has only a `pull_request`
run).

**Not changed:** the other inline shell in `deploy.yml` (head check, VM
reset, service wait). It can move to scripts in the service audit.

I can't confirm why GitHub missed the push. The commit message has no
skip marker, and every push workflow is missing, so it looks like a
GitHub delivery gap, not this repo's config.

<!-- junior-request-attribution:start -->
via @dcramer.
<!-- junior-request-attribution:end -->

<!-- junior-session-footer:start -->
<!-- junior-conversation-id:slack%3AC0ACFA5JBDX%3A1793830221.324719 -->

--

[View
Session](https://junior-prod.sentry.dev/conversations/slack%3AC0ACFA5JBDX%3A1793830221.324719)
[[Sentry]](https://sentry.sentry.io/explore/conversations/slack%3AC0ACFA5JBDX%3A1793830221.324719/?project=4510944073809921)

<!-- junior-session-footer:end -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant