Remove duplicated hidden text - #1550
Merged
Merged
Conversation
Figures embedded as PDF Form XObjects frequently carry the invisible, clipped text of the manuscript page they were exported from. pdfalto emits that text as ordinary words, so GROBID extracts whole paragraphs and section headings two or three times: in a 2,595-article PMC materials-science corpus, 60% of documents were affected (~9% of paragraphs duplicated); in a 1,943-article PMC life-science corpus, 7%. pdfalto gains a -discardClippedText flag re-enabling xpdf's clip test (kermitt2/pdfalto PR to follow); this change lets GROBID pass it, guarded by a new config key pdf.pdfalto.discardClippedText. Off by default, since stock pdfalto binaries do not know the flag; enabling it on a supporting build is output-identical for documents without hidden clipped text.
…ppedText Built from kermitt2/pdfalto ad13f4e (fix/discard-clipped-text) for lin-64, lin_arm-64, mac-64 and mac_arm-64, so every platform GROBID ships understands -discardClippedText. The pdfalto_server wrapper needs no change: it execs the same binary. With no platform left that rejects the option, the reason for opt-in is gone and pdf.pdfalto.discardClippedText now defaults to true (config default, YAML, and the no-config fallback in GrobidProperties). Set it to false when pointing grobid-home at a pdfalto build that predates the option. Effect: figures embedded as Form XObjects stop contributing the invisible, clipped text of the page they were exported from. On 2,595 PMC materials-science articles, normalised body-text similarity against the JATS reference goes 0.7593 -> 0.9017; on 1,943 life-science articles, 0.8926 -> 0.8963. Documents with no clipped text are unaffected -- output is byte-identical.
lfoppiano
changed the title
lfoppiano
changed the title
This was referenced Aug 27, 2026
lfoppiano
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Figures embedded as PDF Form XObjects frequently carry the invisible, clipped text of the manuscript page they were exported from. pdfalto emits that text as ordinary words, so GROBID extracts whole paragraphs and section headings two or three times: in a 2,595-article PMC materials-science corpus, 60% of documents were affected (~9% of paragraphs duplicated); in a 1,943-article PMC life-science corpus, 7%.
pdfalto 0.6.3+ or 0.7.0 (kermitt2/pdfalto#254) will ship a -discardClippedText flag re-enabling xpdf's clip test (kermitt2/pdfalto PR to follow); this change lets GROBID pass it, guarded by a new config key pdf.pdfalto.discardClippedText. Off by default, since stock pdfalto binaries do not know the flag; enabling it on a supporting build is output-identical for documents without hidden clipped text.