Skip to content

feat(bindings): expose mcid and struct-tree role on positioned text items - #446

Open
mathurshubham wants to merge 2 commits into
firecrawl:mainfrom
mathurshubham:fix/expose-mcid-struct-role
Open

feat(bindings): expose mcid and struct-tree role on positioned text items #446
mathurshubham wants to merge 2 commits into
firecrawl:mainfrom
mathurshubham:fix/expose-mcid-struct-role

Conversation

@mathurshubham

@mathurshubham mathurshubham commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Implements the "cheapest version" of #215: surface the structure-tree data the crate already parses, so callers of the positioned-text APIs can separate footnotes, running headers, and marginalia from body text.

Problem

Extracted markdown merges body text with footnotes, running headers, and marginalia into one stream. As #215 lays out, for scholarly and legal documents that is an attribution error, not a formatting one. The building blocks to fix it downstream already exist in the crate — TextItem carries mcid, and the struct tree is parsed internally — but the napi and Python bindings expose neither the resolved role nor a way to get it without a manual (page, mcid) join against extract_structure_elements.

Change

  • Core (Rust): new extract_text_with_positions_with_roles_mem(buffer, pages) (and a path-based wrapper) returning Vec<(TextItem, Option<String>)>. It runs the existing positioned-text extraction, builds a (page, mcid) → role map from the already-parsed /StructTreeRoot (via extract_structure_elements_mem, which resolves custom tags through the document's role map), and pairs each item with its resolved structure type name — "H1""H6", "P", "Note", "Caption", "Figure", etc.
  • napi: TextItem.role: string | null, populated by extractTextWithPositions.
  • Python: TextItem.role: Optional[str], populated by both extract_text_with_positions and extract_text_with_positions_bytes; pdf_inspector.pyi updated.
  • role is None/null when the PDF is untagged, the page has no structure tree, or the item is not part of marked content — so untagged documents behave exactly as before.
  • wasm is unchanged: it exposes no positioned-text item type, so there is no equivalent field to add.

Both bindings now route their positioned-text calls through the single core function, which also removed some duplicated page-set plumbing.

Tests

  • test_positioned_items_carry_resolved_role_tagged — builds a synthetic tagged PDF in-test with /P body text (MCID 0) and a /Note footnote (MCID 1); asserts items carry the right mcid, resolve to roles P and Note, and are separable by role.
  • test_positioned_items_role_none_untagged — untagged PDF yields role = None for every item.
  • Binding-level assertions added to napi/test.mjs and tests/test_python.py.

cargo fmt, cargo clippy -- -D warnings (plus --features ocr), and the full cargo test suite (1015 lib + 167 integration + 2 doc-tests) all pass; the napi crate and --features python builds compile clean.

Out of scope

The heuristic classification for untagged documents discussed in #215 (font-size clustering, position bands, separator rules) is deliberately not included — this PR is the expose-what-we-already-parse slice, which #215 calls out as the cheapest first step. The heuristic layer can build on this API.

Closes #215.


Summary by cubic

Expose resolved structure-tree roles on positioned text items so callers can separate body text from footnotes, running headers, and marginalia. Previously, callers had to join (page, mcid) against structure elements; now each item includes a resolved role, with untagged PDFs behaving the same.

  • Core (Rust): add extract_text_with_positions_with_roles_mem and a path wrapper returning Vec<(TextItem, Option<String>)>. Resolve (page, mcid) -> role from /StructTreeRoot, now parsing the document once and sharing it via structure_elements_from_doc to avoid a second load. pages remains 1-indexed.
  • napi: add TextItem.role: string | null; extractTextWithPositions now routes through the roles API. Untagged PDFs yield role === undefined.
  • Python (pdf_inspector): add TextItem.role: Optional[str]; both extract_text_with_positions and _bytes route through the roles API; pdf_inspector.pyi updated.
  • Untagged PDFs: role is null/None/undefined when untagged, a page lacks a structure tree, or an item isn’t marked; other fields and ordering are unchanged.
  • wasm: unchanged.
  • Tests: core integration tests assert mcid and roles (P, Note) and separability by role; untagged PDFs yield role=None. Binding tests added and tightened for JS undefined.

Migration

  • No breaking changes; consumers may read role when present.

Written for commit 59bb89c. Summary will update on new commits.

Review in cubic

…tems

Extracted text merges body copy with footnotes, running headers, and
marginalia. For tagged PDFs the structure tree already labels each region
(/Note, /Caption, /H1, ...), and TextItem.mcid is the link back to it, but
the bindings only surfaced mcid — callers had to join it against
extract_structure_elements themselves to recover the role.

Surface the resolved role directly on the positioned text item so a caller
can separate the editor's footnote from the author's body text without the
manual (page, mcid) join:

- core: add extract_text_with_positions_with_roles_mem / _with_roles, which
  return each TextItem paired with its resolved structure type name
  (Option<String>). Role is resolved from the already-parsed struct tree via
  the item's mcid; None when the PDF is untagged, the page has no struct
  tree, or the item is not in marked content.
- napi: add TextItem.role (Option<String>); extractTextWithPositions now
  routes through the core roles API.
- python: add TextItem.role (Optional[str]) and update pdf_inspector.pyi;
  both extract_text_with_positions and _bytes route through the core API.

wasm is unchanged: it exposes no positioned-text item, so there is no
equivalent field to add. The heuristic fallback for untagged documents
(font-size / vertical-position / separator-rule / reference-marker
clustering) is intentionally out of scope for this cheapest-first slice.

Tests: synthetic tagged PDF with /P body text and a /Note footnote asserts
mcid and the resolved role come through the core Rust API and are separable;
an untagged PDF yields role=None for every item. Binding-level assertions
added to napi/test.mjs and tests/test_python.py.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 7 files

Shadow auto-approve: would not auto-approve because issues were found.

Fix all with cubic | Re-trigger cubic

Comment thread src/lib.rs Outdated
Comment thread napi/test.mjs Outdated
Address review feedback on the role-join API: extract_text_with_positions_with_roles_mem
previously loaded and parsed the PDF twice (once for positioned text, once inside
extract_structure_elements_mem). The document is now loaded once and feeds both the
positioned-text walk and the struct-tree walk, via a shared structure_elements_from_doc
helper. Also tighten the napi untagged-PDF assertion to pin role === undefined.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 2 files (changes from recent commits).

Shadow auto-approve: would auto-approve. Exposes resolved structure-tree roles (e.g., H1, P, Note) on positioned text items across Rust, Python, and Napi bindings. It consolidates extraction logic to avoid redundant parsing and adds comprehensive integration tests.

Re-trigger cubic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature: label footnote / header / marginal text as distinct regions

1 participant