Repository navigation
fix(secrets): Check escapes, fail closed on hidden fields, learn \/ - #47
Merged
Merged
Conversation
Each string with an escape that is not valid made JSON.parse throw, and a throw costs about 4 us. With a one-character password each such string was decoded, so 8 MiB of `"\<<v>>"` blocked the event loop for 5.9 s. scanStrings now checks each escape as it reads it, and marks a string with a bad escape or a control character, so nothing throws. The walk of the body also skips the strings of lists. A value pattern that takes a quote away can put a credential field outside of a string in the text of the key. replaceStrings now counts the matches of each value. When the text has one fewer times than the body has fields, the recorder does not write the sample, and logs the key. A minted token with `/` is also learned as JSON writes it with `\/`, so a copy in the same body is redacted, as on main. Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com>
dcramer
marked this pull request as ready for review
November 5, 2026 07:02
The count check of a sample compared counts by value, not by place. A value in a field that is not a credential, in a key of an object, or in a list, or two shifted quotes, let a password through. redactRequestWith now looks for each credential field value in its result, as it is, as JSON writes it, and with `\/`. When it finds one, or when a body has more than 64 values, the recorder writes a sample with no text (`withheld`). The sample takes the place of an older sample of the same place, so a later miss does not compare with a stale request. A form body gets the same check. scanStrings now counts the length of the value of each string, and decodeString decodes escapes without JSON.parse. replaceStrings decodes only a string with the length of a credential value. A compare takes the place of a set lookup for each character. The `\/` forms are learned only for minted values. A learned value, such as a SigV4 Authorization header, gets no more parts toward the limit, as on main. Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit eec80c2. Configure here.
The leftover-secret check searched the text for each credential value with `includes`. A long value hit the slow substring path, so one 2 KiB password in an 8 MiB body stalled the event loop for 11 s. It also missed a value that a pattern changed only in part. - Replace the value at the place of each credential field, and each other string with the same value, in the one scan. - Then parse the body of the text again. Withhold the sample when it is not JSON, when a credential field holds other text, when a string is a credential value, or when a field is not `<<redacted>>` at its place in the request. Do the same for form bodies, by count. Drop the cap of 64 values. - Learn the `\/` form of each credential part again, without a count toward the limit, so an echo with `\/` is redacted and SigV4 headroom stays at 510/8,190. - Make `isHex` accept only 0-9, a-f, and A-F. Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com>
…e pass The form check counted `<<redacted>>` fields in every line after the URL, so a key header could take the place of a field that a value pattern took away. It also decoded each part three times, so a 30 MiB form stalled the event loop for 7 s (main: 2.6 s). - Pass the key headers and value names of the rule with the request, and check only the body, after the key header lines. JSON and forms. - Write no sample text when a value name has `"`, `\`, `&`, `=`, `<`, `>`, or a control character: its placeholder can add a field. - Redact a form body in one pass that also counts, skip the decode of a part without `%` or `+`, keep the kinds of field names, and make the set of credential values only when a length matches. - Learn the `\/` forms of a whole value when its `/` is only in parts shorter than 12 characters. They do not count toward the limit. Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com>
…ential field A body that is not JSON or a form, such as JSON with a comma at the end, JSON lines, or a multipart body, skipped the redaction of credential fields, so a password could go into the request sample. The proxy cannot find the fields of such a body. So the sample now has no text when the body has the name of a credential field, in any case, with `_`, `-`, or nothing between its words. This is one search of the body with a regular expression. On a 30 MiB body that is not JSON, it adds about 70 to 240 ms. Keys do not change. Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com>
Member
|
Needs rebased |
Conflicts in README.md and packages/service/src/recorder.ts. Keep the withheld-sample note of this branch and the main-thread redaction of write-token runs from main. The main-thread path now also passes the rule to `redactRequest`, so it checks only the body.
This was referenced Nov 5, 2026
dcramer
added a commit
that referenced
this pull request
Nov 5, 2026
CI already runs on push to `main` (`ci.yml` has `push: branches: [main, "release/**"]`). But GitHub started no push workflows for `09f6582` (the #47 merge): no CI, no Image, no CodeQL. The Deploy run of `9b2ca6a` (#49) then saw a newer head and stopped ("main is at 09f6582 now, not 9b2ca6a. Nothing deploys."). So the head of `main` has no CI, no image and no deploy, and `:main` still points at `6c64d29` (#48). Nothing could start those runs again: neither workflow had `workflow_dispatch`. **Changes** - `ci.yml` and `image.yml`: add `workflow_dispatch`. - `image.yml`: push `:<sha>` on every event except `pull_request`, so a manual run on `main` makes the image. - `deploy.yml`: - `promote` accepts an Image run that isn't a `pull_request` (push or manual). The `branches: [main]` filter still limits Deploy to `main`. - "Wait for CI" moves to `.github/scripts/wait-for-ci.sh` and looks for the CI run of the commit with `--branch main`, not `--event push`. A manual CI run counts. A `pull_request` run of the same commit doesn't, because its branch isn't `main`. - `README.md`: "Deploy" says what to run when a push gets no workflows. **After merge:** this merge is a new push, so it should get CI, Image and Deploy normally. If it doesn't, run `gh workflow run ci.yml --ref main` and `gh workflow run image.yml --ref main`. **Checked** - shellcheck (`--shell=sh --enable=check-extra-masked-returns`) and actionlint pass. `oxfmt --check` passes. - The script ran under busybox `ash` with stub `gh`, `timeout` and `sleep`: success after two empty lists, a failed run, a run that never ends (60 polls, then exit 1), a `gh` error, and no arguments (exit 2). - The live query `gh run list --workflow ci.yml --branch main --commit <sha>` finds the push run for `9b2ca6a`, nothing for `09f6582`, and nothing for `3f11cb0` (#49's PR head, which has only a `pull_request` run). **Not changed:** the other inline shell in `deploy.yml` (head check, VM reset, service wait). It can move to scripts in the service audit. I can't confirm why GitHub missed the push. The commit message has no skip marker, and every push workflow is missing, so it looks like a GitHub delivery gap, not this repo's config. <!-- junior-request-attribution:start --> via @dcramer. <!-- junior-request-attribution:end --> <!-- junior-session-footer:start --> <!-- junior-conversation-id:slack%3AC0ACFA5JBDX%3A1793830221.324719 --> -- [View Session](https://junior-prod.sentry.dev/conversations/slack%3AC0ACFA5JBDX%3A1793830221.324719) [[Sentry]](https://sentry.sentry.io/explore/conversations/slack%3AC0ACFA5JBDX%3A1793830221.324719/?project=4510944073809921) <!-- junior-session-footer:end -->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Follow-up to #46. #46 merged at 540db57, before the fixes from its 5th review landed. This PR is one commit on top of main (37bde8f).
Changes
A throw for each invalid escape blocked the event loop (high).
JSON.parsethrow insidedecodeString, and each throw costs about 4 µs. A one-characterpasswordmakes every escaped string a candidate for decoding.scanStringschecks each escape as it reads it (\",\\,\/,\b,\f,\n,\r,\t, or\uplus four hex digits). It also flags raw control characters, such as a tab in a header line. It marks each string as having no escapes, valid escapes, or a bad one.decodeStringreturnsundefinedfor a bad string and only callsJSON.parseon valid ones, so nothing throws."\\u12"(an escaped backslash, then a plainu12) is not rejected.\/,\u002Dand lone surrogates still redact.credentialValuesnow pushes only objects onto its stack, and loops over lists by index.u0001pattern, event-loop gap"\u12"strings"\x"stringsA pattern that shifts quotes leaked
passwordinto the request sample (medium–high).id-[0-9]+"takes a quote away. In the text of the key, a credential field then sits outside a string, and the sample kept it.replaceStringscounts the matches for each value and compares them with how many credential fields hold that value. If any value has fewer matches than fields, the recorder doesn't write the sample. It logs a line with the key only, and the recording itself is still written.A nested
\/copy of a minted token was stored (low–medium).add()now also learns two more forms of each part that has a/: the\/form, and its JSON-escaped form. The second is the one that matches, because the known-value pass runs overJSON.stringify(recording), where the body's\/appears as\\/./now counts two more parts toward the learn limit.Keys
Recording keys do not change.
request-key.tsis untouched, and the pinned key test (2ee8f643…) passes.Tests
Three new tests in
packages/service/tests/redaction.test.ts. Each one fails on 540db57:u0001-pattern body under an event-loop bound of 3 s (7.0 s on 540db57; 1.16 s here with coverage)\/copy of a minted token in the same bodypnpm checkandpnpm test:coveragepass on Node 24.22.2 (72 tests).Not changed
These were reviewed and accepted as they are: the stall on very large lists (main runs out of memory on the same body), and invalid-JSON bodies with a byte order mark.
via @dcramer.
--
View Session [Sentry]