Repository navigation
Chore: Reduce CPU cost of message ingestion - #771
Conversation
Convert the HTML body to text once for both the search text and snippet, replace the regular expressions in html2text and cleanString with linear scanners, and skip enmime's unused HTML to text down-conversion when storing messages. Stored search text and snippets are unchanged, verified by equivalence and fuzz tests against the previous implementation.
|
Thanks for this. The equivalence and fuzz coverage is impressive, and I reproduced the gains on my own machine (4 cores, server pinned to 2, 100 workers to
|
| size | exploit | control |
|---|---|---|
| 100 KB | 70 ms | 0.8 ms |
| 200 KB | 289 ms | 1.3 ms |
| 400 KB | 1.2 s | 3.5 ms |
| 800 KB | 4.9 s | 5.7 ms |
The old regexp took about 204 ms on the 800 KB case. This is reachable through Store(), ReindexAll() and snippet creation in a default deployment, so a message with a few MB of HTML can pin a core for a long time.
Suggested fix
A tag name is at most about 32 bytes, so cap the scan window:
w := h[i:]
if len(w) > 33 {
w = w[:33]
}
k := strings.IndexByte(w, '>')I tried this on a copy of the branch. The 800 KB case drops to about 6 ms, the existing equivalence tests still pass, and the benchmark and load-test numbers above were measured with the cap applied, so it costs nothing on normal mail.
Test request
Please add a regression test with the exploit shape (strings.Repeat("</", N)) and a matched benign control, with a time or allocation bound. The current equivalence tests compare output only, so they would pass on the quadratic version.
I checked replaceBr and spaceImg and both are linear.
Only search for the closing ">" within the maximum tag name length after each "</". Searching the whole remainder made input with many "</" and no nearby ">" quadratic. Add a regression test comparing adversarial input against a benign control of the same size.
|
Thanks a lot for reproducing the numbers and for catching this. You're right, I confirmed it locally. I've applied your suggested cap and tests. |
Hi,
Thanks again for merging #750. After the index fix I kept load testing the same setup (test traffic through
/api/v1/send, a lot of it), and the next bottleneck turned out to be CPU. Most of it was going into turning the HTML body into text, and that happened three times for every stored message.This PR makes that cheaper without changing anything that gets stored: the search text and snippet are the same as before, byte for byte. I added equivalence tests and fuzz tests against the old code to prove that.
Details and numbers are below. Happy to change anything, split it further, or drop parts you don't like.
Summary
For a message with an HTML body,
Store()currently converts HTML to text three times:createSearchText→html2text.Strip(env.HTML, true)tools.CreateSnippet→html2text.Strip(html, false)env.Text.Store()only needs that text in a rare fallback, so in most cases it is thrown away.html2text.Stripalso runs three regexps over the raw HTML before parsing, plus one more inclean().A CPU profile we took on a 2 CPU box with a ~14KB HTML email showed roughly: HTML→text (both strips) ~37%, enmime parse including its own conversion ~20%, the
cleanStringregexp ~11%, SQLite 5–9%.Changes:
internal/html2text: replaced the regexps with small linear scanners (spaceBlockTags,replaceBr,spaceImg) and a single-passclean(). AddedStripBoth, which returns the text with and without links from a single parse, andStripTrimmedAndFull, which returns exactly what the snippet and the search text used to get (see the note below).internal/storage:Store()andReindexAll()use the newsearchTextAndSnippet(), which parses the HTML once for both values.Store()uses a parser withenmime.DisableTextConversion(true). In the rare case the old code needed enmime's converted text (an HTML part that strips to nothing, such as an image-only email),envelopeText()runs the same converter enmime uses (inbucket/html2text), so the result doesn't change. The message view still uses the normalenvelopeParser.cleanStringusesstrings.Mapinstead of a regexp.internal/mimeparser:New(opts ...enmime.Option), so the store parser can add the option on top of the usual defaults.internal/tools:SnippetFromStripped(), reused byCreateSnippet.Note on whitespace: the snippet was built from the trimmed HTML, but the search text from the untrimmed HTML. In a few edge cases that gives different text. For example, trailing whitespace after an unclosed
<a>makes the HTML parser clone the link, and a text node that is exactlylinkortitleis only skipped when nothing follows it.StripTrimmedAndFullparses the untrimmed HTML once and parses a second time only when one of those cases is possible, so both values stay exactly as before.Related issues
Follow-up to #750 (same load test).
Type of change
Test plan
go test ./...passes.TestStripEquivalentToRegexp: 200k random HTML fragments plus the existing 25KB test email. Each one must match the old regexp implementation exactly, forStrip(with and without links),StripTrimmedAndFulland every helper. I also ran it locally with 19 more seeds (about 3.8M inputs), all equal.FuzzStripEquivalentToRegexp: same check, fuzzed for several minutes.TestSearchTextAndSnippetEquivalent/FuzzSearchTextAndSnippetEquivalent: the old full parse and old functions vs the new store path, on allinternal/storage/testdata/*.emlfiles plus HTML-only edge cases (whitespace-only HTML, image-only HTML, broken trailing markup,link</a>, multipart). SearchText and Snippet must be identical.FuzzCleanStringEquivalent: newcleanStringvs the old regexp version.Performance tests
Setup
CGO_ENABLED=0 go build.developat 5c18887. After = this branch.MP_MAX_MESSAGES=0, no webhook, no UI connected.GOMAXPROCS=2andGOMAXPROCS=8, to see both a small container and a full machine./api/v1/send. 1,000 warm-up messages, then 20,000 measured. CPU per message is the process CPU time over the run divided by 20,000.htmlTestDataininternal/html2text. "Small" is<p>Your code is <b>123456</b></p>.Results
Small messages gain little, as expected, since there is very little HTML to convert. Real HTML emails are where it matters.
Micro benchmark (
go test -run=^$ -bench=StripBoth -benchmem ./internal/html2text/): both conversions on the 25KB email.StripBoth)Reproduce
The client is the same as in #750, with
HTMLset to the contents ofhtmlTestData(or the small body) and an error count added. I can paste it here if that helps.AI assistance disclosure
I used Claude Code to profile the ingest path, write the regexp-free scanners and the equivalence and fuzz tests, and run the benchmarks. The fuzz and random tests are how the whitespace edge cases above were found. I reviewed the code, the equivalence approach and the benchmark results.
Checklist