Skip to content

Chore: Reduce CPU cost of message ingestion - #771

Merged
axllent merged 2 commits into
axllent:developfrom
rurbanovich:perf/ingest-cpu
Nov 14, 2026
Merged

axllent merged 2 commits into
axllent:developfrom
rurbanovich:perf/ingest-cpu

Conversation

@rurbanovich

Copy link
Copy Markdown
Contributor

Hi,

Thanks again for merging #750. After the index fix I kept load testing the same setup (test traffic through /api/v1/send, a lot of it), and the next bottleneck turned out to be CPU. Most of it was going into turning the HTML body into text, and that happened three times for every stored message.

This PR makes that cheaper without changing anything that gets stored: the search text and snippet are the same as before, byte for byte. I added equivalence tests and fuzz tests against the old code to prove that.

Details and numbers are below. Happy to change anything, split it further, or drop parts you don't like.

Summary

For a message with an HTML body, Store() currently converts HTML to text three times:

  1. createSearchText → html2text.Strip(env.HTML, true)
  2. tools.CreateSnippet → html2text.Strip(html, false)
  3. inside enmime, which down-converts HTML-only messages to env.Text. Store() only needs that text in a rare fallback, so in most cases it is thrown away.

html2text.Strip also runs three regexps over the raw HTML before parsing, plus one more in clean().

A CPU profile we took on a 2 CPU box with a ~14KB HTML email showed roughly: HTML→text (both strips) ~37%, enmime parse including its own conversion ~20%, the cleanString regexp ~11%, SQLite 5–9%.

Changes:

  • internal/html2text: replaced the regexps with small linear scanners (spaceBlockTags, replaceBr, spaceImg) and a single-pass clean(). Added StripBoth, which returns the text with and without links from a single parse, and StripTrimmedAndFull, which returns exactly what the snippet and the search text used to get (see the note below).
  • internal/storage: Store() and ReindexAll() use the new searchTextAndSnippet(), which parses the HTML once for both values. Store() uses a parser with enmime.DisableTextConversion(true). In the rare case the old code needed enmime's converted text (an HTML part that strips to nothing, such as an image-only email), envelopeText() runs the same converter enmime uses (inbucket/html2text), so the result doesn't change. The message view still uses the normal envelopeParser. cleanString uses strings.Map instead of a regexp.
  • internal/mimeparser: New(opts ...enmime.Option), so the store parser can add the option on top of the usual defaults.
  • internal/tools: SnippetFromStripped(), reused by CreateSnippet.

Note on whitespace: the snippet was built from the trimmed HTML, but the search text from the untrimmed HTML. In a few edge cases that gives different text. For example, trailing whitespace after an unclosed <a> makes the HTML parser clone the link, and a text node that is exactly link or title is only skipped when nothing follows it. StripTrimmedAndFull parses the untrimmed HTML once and parses a second time only when one of those cases is possible, so both values stay exactly as before.

Related issues

Follow-up to #750 (same load test).

Type of change

  • Bug fix (performance)
  • New feature
  • Refactor / code cleanup
  • Documentation
  • Other (please describe):

Test plan

  • go test ./... passes.
  • TestStripEquivalentToRegexp: 200k random HTML fragments plus the existing 25KB test email. Each one must match the old regexp implementation exactly, for Strip (with and without links), StripTrimmedAndFull and every helper. I also ran it locally with 19 more seeds (about 3.8M inputs), all equal.
  • FuzzStripEquivalentToRegexp: same check, fuzzed for several minutes.
  • TestSearchTextAndSnippetEquivalent / FuzzSearchTextAndSnippetEquivalent: the old full parse and old functions vs the new store path, on all internal/storage/testdata/*.eml files plus HTML-only edge cases (whitespace-only HTML, image-only HTML, broken trailing markup, link</a>, multipart). SearchText and Snippet must be identical.
  • FuzzCleanStringEquivalent: new cleanString vs the old regexp version.
  • End-to-end throughput, before and after (below).

Performance tests

Setup

  • Apple M1 Pro (8 cores), macOS 27, go1.27.4, CGO_ENABLED=0 go build.
  • Before = develop at 5c18887. After = this branch.
  • Fresh SQLite DB on local SSD for every run, MP_MAX_MESSAGES=0, no webhook, no UI connected.
  • GOMAXPROCS=2 and GOMAXPROCS=8, to see both a small container and a full machine.
  • Client: 100 concurrent workers POSTing to /api/v1/send. 1,000 warm-up messages, then 20,000 measured. CPU per message is the process CPU time over the run divided by 20,000.
  • Payloads: "HTML" is the 25KB newsletter already used as htmlTestData in internal/html2text. "Small" is <p>Your code is <b>123456</b></p>.

Results

Cores Payload Before msg/s After msg/s CPU / msg before → after p95 before → after
2 HTML 936–961 1,581–1,626 (1.7×) 2.0 → 1.15 ms 304 → 177 ms
8 HTML 1,379–1,418 2,725–2,834 (2.0×) 2.45 → 1.48 ms 208 → 103 ms
2 Small ~3,200 ~3,290 (+3%) 0.49 → 0.47 ms 91 → 90 ms
8 Small ~3,790 ~3,945 (+4%) 0.70 → 0.67 ms 78 → 74 ms

Small messages gain little, as expected, since there is very little HTML to convert. Real HTML emails are where it matters.

Micro benchmark (go test -run=^$ -bench=StripBoth -benchmem ./internal/html2text/): both conversions on the 25KB email.

ns/op B/op allocs/op
old (regexp, 2 strips) ~600k 695k 3,601
new (StripBoth) ~210k (2.9×) 174k (4×) 1,781

Reproduce

MP_DATABASE=./bench.db MP_MAX_MESSAGES=0 GOMAXPROCS=2 \
  ./mailpit --listen 127.0.0.1:18025 --smtp 127.0.0.1:11025 --disable-version-check -q

The client is the same as in #750, with HTML set to the contents of htmlTestData (or the small body) and an error count added. I can paste it here if that helps.

AI assistance disclosure

  • No AI was used
  • Assisted — AI was used for suggestions, explanations, or small snippets, but the code was primarily written and understood by me
  • Extensive — AI generated a significant portion of the code in this PR

I used Claude Code to profile the ingest path, write the regexp-free scanners and the equivalence and fuzz tests, and run the benchmarks. The fuzz and random tests are how the whitespace edge cases above were found. I reviewed the code, the equivalence approach and the benchmark results.

Checklist

  • I have tested my changes locally
  • I have read and understood all code being submitted (including any AI-generated portions)
  • Documentation has been updated where relevant (not applicable: internal change, no new options)

Convert the HTML body to text once for both the search text and snippet,
replace the regular expressions in html2text and cleanString with linear
scanners, and skip enmime's unused HTML to text down-conversion when
storing messages. Stored search text and snippets are unchanged, verified
by equivalence and fuzz tests against the previous implementation.
@axllent

axllent commented Nov 13, 2026

Copy link
Copy Markdown
Owner

Thanks for this. The equivalence and fuzz coverage is impressive, and I reproduced the gains on my own machine (4 cores, server pinned to 2, 100 workers to /api/v1/send, 25 KB HTML email): about 1.75× throughput (440 to 470 msg/s before, 770 to 830 after), roughly half the p95 latency, and about 40% less CPU per message. Small messages were within noise, as you found. I found one problem while reviewing.

spaceBlockTags is quadratic on adversarial input

For every </, it calls strings.IndexByte(h[i:], '>') over the whole remainder of the string, and only afterwards rejects k > 32. Input with many </ and no nearby > therefore costs O(n²). The old regexp was linear.

I measured strings.Repeat("</", n) against a benign control (</p> repeated to the same size):

size exploit control
100 KB 70 ms 0.8 ms
200 KB 289 ms 1.3 ms
400 KB 1.2 s 3.5 ms
800 KB 4.9 s 5.7 ms

The old regexp took about 204 ms on the 800 KB case. This is reachable through Store(), ReindexAll() and snippet creation in a default deployment, so a message with a few MB of HTML can pin a core for a long time.

Suggested fix

A tag name is at most about 32 bytes, so cap the scan window:

w := h[i:]
if len(w) > 33 {
    w = w[:33]
}
k := strings.IndexByte(w, '>')

I tried this on a copy of the branch. The 800 KB case drops to about 6 ms, the existing equivalence tests still pass, and the benchmark and load-test numbers above were measured with the cap applied, so it costs nothing on normal mail.

Test request

Please add a regression test with the exploit shape (strings.Repeat("</", N)) and a matched benign control, with a time or allocation bound. The current equivalence tests compare output only, so they would pass on the quadratic version.

I checked replaceBr and spaceImg and both are linear.

Only search for the closing ">" within the maximum tag name length after
each "</". Searching the whole remainder made input with many "</" and no
nearby ">" quadratic. Add a regression test comparing adversarial input
against a benign control of the same size.
@rurbanovich

Copy link
Copy Markdown
Contributor Author

Thanks a lot for reproducing the numbers and for catching this. You're right, I confirmed it locally. I've applied your suggested cap and tests.

@axllent
axllent merged commit f706f99 into axllent:develop Nov 14, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants