feat(bindings): expose mcid and struct-tree role on positioned text items - #446
Open
mathurshubham wants to merge 2 commits into
Open
feat(bindings): expose mcid and struct-tree role on positioned text items #446mathurshubham wants to merge 2 commits into
mathurshubham wants to merge 2 commits into
Conversation
…tems Extracted text merges body copy with footnotes, running headers, and marginalia. For tagged PDFs the structure tree already labels each region (/Note, /Caption, /H1, ...), and TextItem.mcid is the link back to it, but the bindings only surfaced mcid — callers had to join it against extract_structure_elements themselves to recover the role. Surface the resolved role directly on the positioned text item so a caller can separate the editor's footnote from the author's body text without the manual (page, mcid) join: - core: add extract_text_with_positions_with_roles_mem / _with_roles, which return each TextItem paired with its resolved structure type name (Option<String>). Role is resolved from the already-parsed struct tree via the item's mcid; None when the PDF is untagged, the page has no struct tree, or the item is not in marked content. - napi: add TextItem.role (Option<String>); extractTextWithPositions now routes through the core roles API. - python: add TextItem.role (Optional[str]) and update pdf_inspector.pyi; both extract_text_with_positions and _bytes route through the core API. wasm is unchanged: it exposes no positioned-text item, so there is no equivalent field to add. The heuristic fallback for untagged documents (font-size / vertical-position / separator-rule / reference-marker clustering) is intentionally out of scope for this cheapest-first slice. Tests: synthetic tagged PDF with /P body text and a /Note footnote asserts mcid and the resolved role come through the core Rust API and are separable; an untagged PDF yields role=None for every item. Binding-level assertions added to napi/test.mjs and tests/test_python.py.
There was a problem hiding this comment.
All reported issues were addressed across 7 files
Shadow auto-approve: would not auto-approve because issues were found.
Fix all with cubic | Re-trigger cubic
Address review feedback on the role-join API: extract_text_with_positions_with_roles_mem previously loaded and parsed the PDF twice (once for positioned text, once inside extract_structure_elements_mem). The document is now loaded once and feeds both the positioned-text walk and the struct-tree walk, via a shared structure_elements_from_doc helper. Also tighten the napi untagged-PDF assertion to pin role === undefined.
There was a problem hiding this comment.
0 issues found across 2 files (changes from recent commits).
Shadow auto-approve: would auto-approve. Exposes resolved structure-tree roles (e.g., H1, P, Note) on positioned text items across Rust, Python, and Napi bindings. It consolidates extraction logic to avoid redundant parsing and adds comprehensive integration tests.
Re-trigger cubic
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the "cheapest version" of #215: surface the structure-tree data the crate already parses, so callers of the positioned-text APIs can separate footnotes, running headers, and marginalia from body text.
Problem
Extracted markdown merges body text with footnotes, running headers, and marginalia into one stream. As #215 lays out, for scholarly and legal documents that is an attribution error, not a formatting one. The building blocks to fix it downstream already exist in the crate —
TextItemcarriesmcid, and the struct tree is parsed internally — but the napi and Python bindings expose neither the resolved role nor a way to get it without a manual(page, mcid)join againstextract_structure_elements.Change
extract_text_with_positions_with_roles_mem(buffer, pages)(and a path-based wrapper) returningVec<(TextItem, Option<String>)>. It runs the existing positioned-text extraction, builds a(page, mcid) → rolemap from the already-parsed/StructTreeRoot(viaextract_structure_elements_mem, which resolves custom tags through the document's role map), and pairs each item with its resolved structure type name —"H1"…"H6","P","Note","Caption","Figure", etc.TextItem.role: string | null, populated byextractTextWithPositions.TextItem.role: Optional[str], populated by bothextract_text_with_positionsandextract_text_with_positions_bytes;pdf_inspector.pyiupdated.roleisNone/nullwhen the PDF is untagged, the page has no structure tree, or the item is not part of marked content — so untagged documents behave exactly as before.Both bindings now route their positioned-text calls through the single core function, which also removed some duplicated page-set plumbing.
Tests
test_positioned_items_carry_resolved_role_tagged— builds a synthetic tagged PDF in-test with/Pbody text (MCID 0) and a/Notefootnote (MCID 1); asserts items carry the right mcid, resolve to rolesPandNote, and are separable by role.test_positioned_items_role_none_untagged— untagged PDF yieldsrole = Nonefor every item.napi/test.mjsandtests/test_python.py.cargo fmt,cargo clippy -- -D warnings(plus--features ocr), and the fullcargo testsuite (1015 lib + 167 integration + 2 doc-tests) all pass; the napi crate and--features pythonbuilds compile clean.Out of scope
The heuristic classification for untagged documents discussed in #215 (font-size clustering, position bands, separator rules) is deliberately not included — this PR is the expose-what-we-already-parse slice, which #215 calls out as the cheapest first step. The heuristic layer can build on this API.
Closes #215.
Summary by cubic
Expose resolved structure-tree roles on positioned text items so callers can separate body text from footnotes, running headers, and marginalia. Previously, callers had to join (page, mcid) against structure elements; now each item includes a resolved role, with untagged PDFs behaving the same.
extract_text_with_positions_with_roles_memand a path wrapper returningVec<(TextItem, Option<String>)>. Resolve(page, mcid) -> rolefrom/StructTreeRoot, now parsing the document once and sharing it viastructure_elements_from_docto avoid a second load.pagesremains 1-indexed.napi: addTextItem.role: string | null;extractTextWithPositionsnow routes through the roles API. Untagged PDFs yieldrole === undefined.pdf_inspector): addTextItem.role: Optional[str]; bothextract_text_with_positionsand_bytesroute through the roles API;pdf_inspector.pyiupdated.roleisnull/None/undefinedwhen untagged, a page lacks a structure tree, or an item isn’t marked; other fields and ordering are unchanged.mcidand roles (P,Note) and separability by role; untagged PDFs yieldrole=None. Binding tests added and tightened for JSundefined.Migration
rolewhen present.Written for commit 59bb89c. Summary will update on new commits.