insert_htmlbox auto base-direction ignores the UAX#9 first-strong rule for mixed RTL+LTR paragraphs

[hrokholskyi]

Versions

• PyMuPDF: 1.28.2
• MuPDF: 1.28.2 (bundled)
• Platform: macOS (Darwin), Python 3.12, arm64

Summary

Page.insert_htmlbox() renders a paragraph that starts with a strong
right-to-left (RTL) character (Arabic) followed by Latin text using an
LTR base direction, instead of the RTL base direction required by
UAX#9 (the Unicode Bidirectional Algorithm).

The visual result is mirrored left-to-right compared to correct
RTL-paragraph layout: the Arabic run appears on the left and the
trailing Latin run appears on the right, when the Arabic run (rightmost)
should lead and the Latin run should trail to its left.

Repro steps

import pymupdf

TEXT = "رقم الموظف / Employee Code"

doc = pymupdf.open()
page = doc.new_page(width=250, height=80)
rect = pymupdf.Rect(10, 10, 210, 60)
page.insert_htmlbox(rect, f"<div>{TEXT}</div>", css="body{font-size:14pt;}")

for w in page.get_text("words"):
    print(w[4], w[0], w[2])   # text, x0, x1

pix = page.get_pixmap(clip=rect, dpi=150)
pix.save("out.png")

The full test (three variants: default, dir="rtl" attribute, CSS
direction: rtl, each dumping word origins and a PNG) reproduces the
numbers shown in the tables below.

Observed vs expected

Default (no dir attribute, no CSS direction), 200pt-wide content box:

word	x0	x1
رقم (raqm)	54.4	73.1
الموظف (al-muwazzaf)	11.0	50.3
/	77.2	84.0
Employee	88.1	148.9
Code	153.0	184.1

The Arabic run occupies the left side of the box (x approx 11-73) and the
Latin run occupies the right side (x approx 77-184). Visually the line
renders as:

رقم الموظف / Employee Code

(Arabic on the left, Latin on the right.)

This is an LTR base-direction layout. Per UAX#9, a paragraph whose first
strong directional character is RTL (here, the first Arabic letter)
must use an RTL base direction. The correct layout mirrors the above,
placing the Arabic run at the right edge of the box and the trailing
Latin run to its left, i.e. visually:

Employee Code / رقم الموظف

Workaround found (this is an auto-detection issue, not a hard layout bug)

Explicitly setting a base direction fixes the output. Both of the
following give the identical, correct result:

• HTML attribute: <div dir="rtl">...</div>
• CSS: div{direction:rtl;}

With either applied, the same input produces:

word	x0	x1
رقم	190.3	209.0
الموظف	146.9	186.2
/	135.9	142.8
Employee	35.8	96.5
Code	100.6	131.7

Arabic now sits at the right edge (x approx 146-209) and the Latin run
sits at the left (x approx 35-131), which is the expected mirrored,
RTL-base layout.

This narrows the bug down to auto-detection specifically:
insert_htmlbox()'s automatic base-direction detection does not honor
the first-strong-character rule from UAX#9 P2/P3 when no explicit
dir/direction is given. The underlying bidi reordering and mirroring
logic itself is correct once the base direction is supplied explicitly;
only the auto-detection step picks the wrong default.

Impact

Any pipeline that generates HTML fragments programmatically (for
example, overlaying translated or bilingual labels onto a page image or
a form) and does not always know in advance whether a given string will
start with an RTL character produces visibly wrong output for
mixed-direction strings: the RTL portion of the label swaps sides with
the LTR portion. Because the failure only shows up for specific string
content (RTL-first, mixed-script) rather than for a whole document, it
typically passes casual testing with pure-LTR or pure-RTL sample
strings and only surfaces once real mixed-language content is rendered.
The workaround (explicitly setting dir="rtl" or CSS direction: rtl
whenever the first strong character of the string is RTL) avoids the bug
but requires the caller to run its own first-strong-character detection
before calling insert_htmlbox(), which the API should be doing
internally.

Suggested fix direction

In the HTML/CSS layout path used by insert_htmlbox(), when no explicit
dir attribute or CSS direction property is present, the paragraph's
base direction should be determined by scanning for the first character
with strong directionality (UAX#9 rule P2/P3) rather than defaulting to
LTR.