Skip to content

fix(secrets): Check escapes, fail closed on hidden fields, learn \/ - #47

Merged
dcramer merged 6 commits into
mainfrom
fix/redaction-gaps-2
Nov 5, 2026
Merged

dcramer merged 6 commits into
mainfrom
fix/redaction-gaps-2

Conversation

@sentry-junior

@sentry-junior sentry-junior Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Follow-up to #46. #46 merged at 540db57, before the fixes from its 5th review landed. This PR is one commit on top of main (37bde8f).

Changes

A throw for each invalid escape blocked the event loop (high).

  • Before: each JSON string with an invalid escape made JSON.parse throw inside decodeString, and each throw costs about 4 µs. A one-character password makes every escaped string a candidate for decoding.
  • Now: scanStrings checks each escape as it reads it (\", \\, \/, \b, \f, \n, \r, \t, or \u plus four hex digits). It also flags raw control characters, such as a tab in a header line. It marks each string as having no escapes, valid escapes, or a bad one. decodeString returns undefined for a bad string and only calls JSON.parse on valid ones, so nothing throws.
  • The check walks escape pairs from left to right, so a valid "\\u12" (an escaped backslash, then a plain u12) is not rejected. \/, \u002D and lone surrogates still redact.
  • credentialValues now pushes only objects onto its stack, and loops over lists by index.
Case 540db57 This PR
8 MiB body, u0001 pattern, event-loop gap 5.9 s about 300 ms (3ea94be: 758 ms)
47 MiB of "\u12" strings 38.7 s 1.7 s
33 MiB of "\x" strings 37.8 s 1.15 s

A pattern that shifts quotes leaked password into the request sample (medium–high).

  • Before: a value pattern such as id-[0-9]+" takes a quote away. In the text of the key, a credential field then sits outside a string, and the sample kept it.
  • Now: replaceStrings counts the matches for each value and compares them with how many credential fields hold that value. If any value has fewer matches than fields, the recorder doesn't write the sample. It logs a line with the key only, and the recording itself is still written.
  • In the worker, miss text that can't be redacted takes the same path as a time-limit failure. The README notes the new behavior.

A nested \/ copy of a minted token was stored (low–medium).

  • add() now also learns two more forms of each part that has a /: the \/ form, and its JSON-escaped form. The second is the one that matches, because the known-value pass runs over JSON.stringify(recording), where the body's \/ appears as \\/.
  • A learned header or query value with a / now counts two more parts toward the learn limit.

Keys

Recording keys do not change. request-key.ts is untouched, and the pinned key test (2ee8f643…) passes.

Tests

Three new tests in packages/service/tests/redaction.test.ts. Each one fails on 540db57:

  • an 8 MiB u0001-pattern body under an event-loop bound of 3 s (7.0 s on 540db57; 1.16 s here with coverage)
  • a pattern that hides a credential field, so the sample is not written
  • a nested \/ copy of a minted token in the same body

pnpm check and pnpm test:coverage pass on Node 24.22.2 (72 tests).

Not changed

These were reviewed and accepted as they are: the stall on very large lists (main runs out of memory on the same body), and invalid-JSON bodies with a byte order mark.

via @dcramer.

--

View Session [Sentry]

Each string with an escape that is not valid made JSON.parse throw, and a
throw costs about 4 us. With a one-character password each such string
was decoded, so 8 MiB of `"\<<v>>"` blocked the event loop for 5.9 s.
scanStrings now checks each escape as it reads it, and marks a string
with a bad escape or a control character, so nothing throws. The walk of
the body also skips the strings of lists.

A value pattern that takes a quote away can put a credential field
outside of a string in the text of the key. replaceStrings now counts the
matches of each value. When the text has one fewer times than the body
has fields, the recorder does not write the sample, and logs the key.

A minted token with `/` is also learned as JSON writes it with `\/`, so
a copy in the same body is redacted, as on main.

Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com>
@dcramer
dcramer marked this pull request as ready for review November 5, 2026 07:02
The count check of a sample compared counts by value, not by place. A
value in a field that is not a credential, in a key of an object, or in
a list, or two shifted quotes, let a password through. redactRequestWith
now looks for each credential field value in its result, as it is, as
JSON writes it, and with `\/`. When it finds one, or when a body has
more than 64 values, the recorder writes a sample with no text
(`withheld`). The sample takes the place of an older sample of the same
place, so a later miss does not compare with a stale request. A form
body gets the same check.

scanStrings now counts the length of the value of each string, and
decodeString decodes escapes without JSON.parse. replaceStrings decodes
only a string with the length of a credential value. A compare takes
the place of a set lookup for each character.

The `\/` forms are learned only for minted values. A learned value, such
as a SigV4 Authorization header, gets no more parts toward the limit, as
on main.

Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit eec80c2. Configure here.

Comment thread packages/service/src/secrets.ts Resolved
sentry-junior Bot and others added 2 commits November 5, 2026 08:21
The leftover-secret check searched the text for each credential value
with `includes`. A long value hit the slow substring path, so one 2 KiB
password in an 8 MiB body stalled the event loop for 11 s. It also
missed a value that a pattern changed only in part.

- Replace the value at the place of each credential field, and each
  other string with the same value, in the one scan.
- Then parse the body of the text again. Withhold the sample when it is
  not JSON, when a credential field holds other text, when a string is
  a credential value, or when a field is not `<<redacted>>` at its place
  in the request. Do the same for form bodies, by count. Drop the cap of
  64 values.
- Learn the `\/` form of each credential part again, without a count
  toward the limit, so an echo with `\/` is redacted and SigV4 headroom
  stays at 510/8,190.
- Make `isHex` accept only 0-9, a-f, and A-F.

Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com>
…e pass

The form check counted `<<redacted>>` fields in every line after the URL,
so a key header could take the place of a field that a value pattern
took away. It also decoded each part three times, so a 30 MiB form
stalled the event loop for 7 s (main: 2.6 s).

- Pass the key headers and value names of the rule with the request,
  and check only the body, after the key header lines. JSON and forms.
- Write no sample text when a value name has `"`, `\`, `&`, `=`, `<`,
  `>`, or a control character: its placeholder can add a field.
- Redact a form body in one pass that also counts, skip the decode of a
  part without `%` or `+`, keep the kinds of field names, and make the
  set of credential values only when a length matches.
- Learn the `\/` forms of a whole value when its `/` is only in parts
  shorter than 12 characters. They do not count toward the limit.

Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com>
Comment thread packages/service/src/secrets.ts Resolved
…ential field

A body that is not JSON or a form, such as JSON with a comma at the end,
JSON lines, or a multipart body, skipped the redaction of credential
fields, so a password could go into the request sample. The proxy cannot
find the fields of such a body. So the sample now has no text when the
body has the name of a credential field, in any case, with `_`, `-`, or
nothing between its words. This is one search of the body with a regular
expression. On a 30 MiB body that is not JSON, it adds about 70 to 240 ms.

Keys do not change.

Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com>
@dcramer

dcramer commented Nov 5, 2026

Copy link
Copy Markdown
Member

Needs rebased

Conflicts in README.md and packages/service/src/recorder.ts. Keep the
withheld-sample note of this branch and the main-thread redaction of
write-token runs from main. The main-thread path now also passes the
rule to `redactRequest`, so it checks only the body.
@dcramer
dcramer merged commit 09f6582 into main Nov 5, 2026
16 checks passed
dcramer added a commit that referenced this pull request Nov 5, 2026
CI already runs on push to `main` (`ci.yml` has `push: branches: [main,
"release/**"]`). But GitHub started no push workflows for `09f6582` (the
#47 merge): no CI, no Image, no CodeQL. The Deploy run of `9b2ca6a`
(#49) then saw a newer head and stopped ("main is at 09f6582 now, not
9b2ca6a. Nothing deploys."). So the head of `main` has no CI, no image
and no deploy, and `:main` still points at `6c64d29` (#48). Nothing
could start those runs again: neither workflow had `workflow_dispatch`.

**Changes**
- `ci.yml` and `image.yml`: add `workflow_dispatch`.
- `image.yml`: push `:<sha>` on every event except `pull_request`, so a
manual run on `main` makes the image.
- `deploy.yml`:
- `promote` accepts an Image run that isn't a `pull_request` (push or
manual). The `branches: [main]` filter still limits Deploy to `main`.
- "Wait for CI" moves to `.github/scripts/wait-for-ci.sh` and looks for
the CI run of the commit with `--branch main`, not `--event push`. A
manual CI run counts. A `pull_request` run of the same commit doesn't,
because its branch isn't `main`.
- `README.md`: "Deploy" says what to run when a push gets no workflows.

**After merge:** this merge is a new push, so it should get CI, Image
and Deploy normally. If it doesn't, run `gh workflow run ci.yml --ref
main` and `gh workflow run image.yml --ref main`.

**Checked**
- shellcheck (`--shell=sh --enable=check-extra-masked-returns`) and
actionlint pass. `oxfmt --check` passes.
- The script ran under busybox `ash` with stub `gh`, `timeout` and
`sleep`: success after two empty lists, a failed run, a run that never
ends (60 polls, then exit 1), a `gh` error, and no arguments (exit 2).
- The live query `gh run list --workflow ci.yml --branch main --commit
<sha>` finds the push run for `9b2ca6a`, nothing for `09f6582`, and
nothing for `3f11cb0` (#49's PR head, which has only a `pull_request`
run).

**Not changed:** the other inline shell in `deploy.yml` (head check, VM
reset, service wait). It can move to scripts in the service audit.

I can't confirm why GitHub missed the push. The commit message has no
skip marker, and every push workflow is missing, so it looks like a
GitHub delivery gap, not this repo's config.

<!-- junior-request-attribution:start -->
via @dcramer.
<!-- junior-request-attribution:end -->

<!-- junior-session-footer:start -->
<!-- junior-conversation-id:slack%3AC0ACFA5JBDX%3A1793830221.324719 -->

--

[View
Session](https://junior-prod.sentry.dev/conversations/slack%3AC0ACFA5JBDX%3A1793830221.324719)
[[Sentry]](https://sentry.sentry.io/explore/conversations/slack%3AC0ACFA5JBDX%3A1793830221.324719/?project=4510944073809921)

<!-- junior-session-footer:end -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant