Skip to content

Fix RFC 2231 continuation parameter decoding (multi-octet split + charset inheritance) - #154

Open
gaoflow wants to merge 1 commit into
stalwartlabs:mainfrom
gaoflow:fix-rfc2231-continuation-decode
Open

Fix RFC 2231 continuation parameter decoding (multi-octet split + charset inheritance)#154
gaoflow wants to merge 1 commit into
stalwartlabs:mainfrom
gaoflow:fix-rfc2231-continuation-decode

Conversation

@gaoflow

@gaoflow gaoflow commented Jul 20, 2026

Copy link
Copy Markdown

RFC 2231 §3/§4.1 lets a long parameter value be split into numbered continuation
sections. The charset'lang' prefix appears only on section 0, later encoded sections
carry raw percent-encoded octets in that same charset, and a multi-octet character may
legally be split across sections. The correct decode is: percent-decode each encoded
section to octets, concatenate all octets, then charset-decode once.

parse_content_type instead decodes each section independently — percent-decode and
charset-decode the section's bytes on their own — and concatenates the resulting strings,
resetting the charset between sections. Two things break.

1. A multi-octet character split across sections decodes each fragment alone, so each
becomes U+FFFD:

Content-Type: application/x-stuff; name*0*=UTF-8''%E2%98; name*1*=%83.txt

got       "\u{FFFD}\u{FFFD}.txt"
expected  "☃.txt"

%E2%98 and %83 are the three UTF-8 bytes of U+2603. Both Go's mime.ParseMediaType
and Python's email (collapse_rfc2231_value) decode this header to ☃.txt.

2. The charset given only on section 0 is not inherited by later standard-form
sections, so their non-ASCII octets fall back to UTF-8:

Content-Type: application/x-stuff; name*0*=ISO-8859-1''%E9; name*1*=%E8.txt

got       "é\u{FFFD}.txt"
expected  "éè.txt"     (Python email agrees)

Fix

Keep each encoded section's percent-decoded octets, resolve the charset from section 0
(still honoring an explicit per-section charset' prefix, so inputs that repeat it decode
as before), concatenate the octets across sections, then charset-decode once. Literal
(non-*) sections are appended verbatim.

New content_type.json cases cover the UTF-8 split above, the section-0-only charset,
out-of-order sections (*1* before *0*, charset still taken from index 0), and an
encoded section followed by a literal one in a non-UTF-8 charset. The existing
continuation tests stay byte-identical — including the non-standard
filename*0*=iso-8859-1'es'… case that repeats the charset on every section, the
title*0*=…; title*2="…" mixed encoded/literal case, and the out-of-order
name*1=…; name*0=…; name*2*=… integration test.

cargo test and cargo test --features full_encoding both pass.

Continuation sections (name*0*=...; name*1*=...) were decoded
independently and the decoded strings concatenated. Per RFC 2231 4.1
the percent-decoded octets of all encoded sections form one byte stream
that must be concatenated before the charset decode, and the
charset'lang' prefix only appears on section 0.

The per-section decode mangled a multi-octet character split across
sections (each fragment became U+FFFD) and dropped the section-0 charset
for later sections, so their non-ASCII octets fell back to UTF-8.

Defer the decode: keep each encoded section's raw octets, inherit the
section-0 charset, concatenate, then decode once. Literal sections are
appended verbatim.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant