Skip to content

test(key-worker): Prove the watchdog without a race with the vm timeout - #52

Merged
dcramer merged 1 commit into
mainfrom
test/key-worker-watchdog
Nov 5, 2026
Merged

dcramer merged 1 commit into
mainfrom
test/key-worker-watchdog

Conversation

@sentry-junior

@sentry-junior sentry-junior Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

"stops a key that the time limit of vm cannot stop" expected the watchdog message. On a fast machine, JSON.parse() of the body ends before the 1 s margin, and the vm timeout fails the key first ("the value patterns took over 1 ms"). That is what failed test on roach#50 (run 42061990306).

  • The test now accepts either message. It still checks that the key ends near its time limit and that the next job works.
  • A new test stops the worker with SIGSTOP, so only the watchdog can end the job on any machine. It checks the watchdog message, the time that the run gets, the time to the error, and that the next job gets a new worker.

Only packages/service/tests/service.test.ts changes.

Checks: pnpm check passes (107 tests). 5 repeated runs pass. A fast-machine simulation fails the old test with the same CI error. With WATCHDOG_MARGIN_MS=20000, the new test fails.

via @dcramer.

--

View Session [Sentry]

"stops a key that the time limit of vm cannot stop" expected the
watchdog message. On a fast machine, `JSON.parse()` of the body ends
before the margin of 1 s, and then the `vm` timeout fails the key first
("the value patterns took over 1 ms"), as on roach#50 CI run
42061990306. A body of 900,000 keys shows it on any machine.

That test now takes either message, and still checks that the key ends
near its time limit and that the next job works. A new test stops the
worker with SIGSTOP, so only the watchdog can end the job on any
machine. It checks the watchdog message, the time that the run gets,
the time to the error, and that the next job gets a new worker. With a
margin of 20 s, it fails.

Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com>
@dcramer
dcramer marked this pull request as ready for review November 5, 2026 16:57
@dcramer
dcramer merged commit db4cfc5 into main Nov 5, 2026
16 checks passed
@dcramer
dcramer deleted the test/key-worker-watchdog branch November 5, 2026 16:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant