Skip to content

fix(parser): recover root-bearing linearized xrefs - #435

Open
yzxcj797 wants to merge 6 commits into
firecrawl:mainfrom
yzxcj797:fix/345-padded-linearized-xref
Open

fix(parser): recover root-bearing linearized xrefs#435
yzxcj797 wants to merge 6 commits into
firecrawl:mainfrom
yzxcj797:fix/345-padded-linearized-xref

Conversation

@yzxcj797

@yzxcj797 yzxcj797 commented Aug 20, 2026

Copy link
Copy Markdown

Problem

Issue #345's scanned PDF reports page_count: 0 and rejects conversion with OCR is required. The file is linearized and has more than 512 bytes of padding after its final %%EOF.

The document has two classic xref tables:

  • an early table whose trailer contains /Root and /Prev;
  • the final table, which completes the object list but has no /Root in its own trailer.

lopdf cannot find the final EOF through the padding, so container repair runs. The old repair selected the newest classic xref without inspecting its trailer. That loads the rootless final table, leaves the catalog unreachable, and produces a zero-page result even though the root-bearing table and its /Prev chain are intact.

Change

  • Enumerate validated classic xref starts from newest to oldest.
  • Prefer the newest table whose own trailer names /Root, which also recovers the rest of the chain through /Prev.
  • Keep the newest valid table as a fallback for unusual trailer layouts.
  • Add a synthetic, stdlib-only linearized PDF regression with two xref tables and 600 trailing NUL bytes. It fails on main with page_count == 0 and now returns one text page containing Hello World.

The sample from the issue was also checked locally: it now reports four scanned pages (still requiring OCR) instead of zero pages.

Tests

  • cargo test --lib — 987 passed.
  • cargo test --test integration_tests test_process_pdf_repairs_padded_linearized_xref_chain -- --nocapture — 1 passed.
  • cargo clippy --lib -- -D warnings — clean.
  • cargo fmt — clean.

Fixes #345


Summary by cubic

Recover linearized PDFs with padded EOF by preferring a root-bearing classic xref and requiring standalone token boundaries for xref/trailer. Previously repair always chose the newest classic xref and could pick a rootless final table, producing page_count: 0; now it selects the newest xref whose trailer has /Root, or keeps the newest rootless table only when its /Prev points backward, preserving revisions and restoring a reachable catalog.

  • Scans classic xref tables newest→oldest; bounds trailer searches to before the next xref; enforces leading/trailing token boundaries for standalone keywords.
  • Parses trailer dictionaries for top‑level keys only; ignores comments, strings, and nested dictionaries; reuses a whitespace/comment scanner; guards short inputs.
  • Retains the newest rootless table only if /Prev points backward; rejects forward /Prev and falls back to the newest root-bearing table; keeps the newest valid table when no root-bearing trailer exists.
  • Adds a stdlib‑only regression PDF with 600 trailing NULs and unit tests for commented “trailer”, nested/commented /Root, backward/forward /Prev, and keyword boundary checks; the sample in Could not convert this file: unsupported input: PDF has no extractable text (Scanned, 0 pages): OCR is required #345 now reports four scanned pages instead of zero.

Fixes #345.

Written for commit 9ddc2a9. Summary will update on new commits.

Review in cubic

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 2 files

Shadow auto-approve: would not auto-approve because issues were found.

Fix all with cubic | Re-trigger cubic

Comment thread src/lib.rs Outdated
Comment thread src/lib.rs
Comment thread src/lib.rs Outdated
Comment thread src/lib.rs Outdated
Guard short inputs, parse trailer dictionaries without mistaking comments or nested keys for /Root, bound trailer searches to the next xref table, and preserve amended /Prev chains.
Only retain the newest rootless table when its top-level /Prev offset points backward, and cover forward malformed chains with a regression test.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 1 file (changes from recent commits).

Shadow auto-approve: would not auto-approve because issues were found.
Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic

Comment thread src/lib.rs
Ignore comment text when locating the standalone trailer keyword and add a regression test for a root-bearing trailer following a commented placeholder.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 1 file (changes from recent commits).

Shadow auto-approve: would not auto-approve because issues were found.
Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic

Comment thread src/lib.rs Outdated

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 1 file (changes from recent commits).

Shadow auto-approve: would not auto-approve because issues were found.
Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic

Comment thread src/lib.rs Outdated

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 1 file (changes from recent commits).

Shadow auto-approve: would auto-approve. Fix bug in xref table selection during repair for linearized PDFs with padded EOF; prefer root-bearing trailer to avoid zero-page result.

Re-trigger cubic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Could not convert this file: unsupported input: PDF has no extractable text (Scanned, 0 pages): OCR is required

1 participant