Fix RFC 2231 continuation parameter decoding (multi-octet split + charset inheritance) - #154
Open
gaoflow wants to merge 1 commit into
Open
Fix RFC 2231 continuation parameter decoding (multi-octet split + charset inheritance)#154gaoflow wants to merge 1 commit into
gaoflow wants to merge 1 commit into
Conversation
Continuation sections (name*0*=...; name*1*=...) were decoded independently and the decoded strings concatenated. Per RFC 2231 4.1 the percent-decoded octets of all encoded sections form one byte stream that must be concatenated before the charset decode, and the charset'lang' prefix only appears on section 0. The per-section decode mangled a multi-octet character split across sections (each fragment became U+FFFD) and dropped the section-0 charset for later sections, so their non-ASCII octets fell back to UTF-8. Defer the decode: keep each encoded section's raw octets, inherit the section-0 charset, concatenate, then decode once. Literal sections are appended verbatim.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
RFC 2231 §3/§4.1 lets a long parameter value be split into numbered continuation
sections. The
charset'lang'prefix appears only on section 0, later encoded sectionscarry raw percent-encoded octets in that same charset, and a multi-octet character may
legally be split across sections. The correct decode is: percent-decode each encoded
section to octets, concatenate all octets, then charset-decode once.
parse_content_typeinstead decodes each section independently — percent-decode andcharset-decode the section's bytes on their own — and concatenates the resulting strings,
resetting the charset between sections. Two things break.
1. A multi-octet character split across sections decodes each fragment alone, so each
becomes U+FFFD:
%E2%98and%83are the three UTF-8 bytes of U+2603. Both Go'smime.ParseMediaTypeand Python's
email(collapse_rfc2231_value) decode this header to☃.txt.2. The charset given only on section 0 is not inherited by later standard-form
sections, so their non-ASCII octets fall back to UTF-8:
Fix
Keep each encoded section's percent-decoded octets, resolve the charset from section 0
(still honoring an explicit per-section
charset'prefix, so inputs that repeat it decodeas before), concatenate the octets across sections, then charset-decode once. Literal
(non-
*) sections are appended verbatim.New
content_type.jsoncases cover the UTF-8 split above, the section-0-only charset,out-of-order sections (
*1*before*0*, charset still taken from index 0), and anencoded section followed by a literal one in a non-UTF-8 charset. The existing
continuation tests stay byte-identical — including the non-standard
filename*0*=iso-8859-1'es'…case that repeats the charset on every section, thetitle*0*=…; title*2="…"mixed encoded/literal case, and the out-of-ordername*1=…; name*0=…; name*2*=…integration test.cargo testandcargo test --features full_encodingboth pass.