Use PDF outline to correct the header hierarchy - #1509
Open
lfoppiano wants to merge 6 commits into
Open
Conversation
The single currentParentId int was reset to -1 at every TOCITEMLIST close, so items following a nested list were reattached to the root instead of to their real parent section. Replace it with a balanced Deque pushed once per TOCITEMLIST (-1 when the list has no idItemParent) and popped on close. Port the SAX handler test to Kotlin with nested and malformed outline fixtures; drop the superseded Java test.
Locate a section head in the outline tree by Ratcliff/Obershelp similarity (>=0.90), returning its depth. Normalizes non-breaking spaces, soft hyphens and '|' number separators that pdfalto outline labels carry but section heads do not.
When the outline shows a head/sub-head hierarchy, keep sub-heads inside the current section's <div> instead of opening a new (empty) <div> per head. A new div opens only at outline-confirmed main-level heads (depth == minDepth) or at heads absent from the outline; the flat div structure is preserved. Activated per text piece only when adjacent section heads exist and the outline is present, so documents without an outline are unchanged. The legacy </head><head> div-splitting cleanup is skipped when this applies and scoped to the region the call appended, since the buffer is shared across body/annex calls. Enable outline extraction for full-text processing.
…ection
The previous rule folded any head deeper than the shallowest matched one into
the open div. When the sequence labeller missed a mid-level heading, the next
section's sub-heads were pulled into the previous section's div. An audit over
10k PDFs found 92 such cross-section divs.
Fold a sub-head only when all of the following hold:
- its outline node is a descendant of the head that opened the div, so a
sub-head whose parent heading was missed starts its own div instead;
- the section numbering does not contradict that nesting, which overrides
degenerate outlines that linearly nest unrelated sections;
- the head could be located in the outline at all.
Match heads against the outline by their labelled text first, then retry with
the section number stripped: outlines commonly store the bare title ("Results")
while the head carries a number ("2. Results"), which a short title pushes below
the similarity threshold and would leave the parent head unanchored.
Measured over a 20% sample (1985 documents) against the same documents: cross
-section divs 7 -> 0, while grouping still removes 4621 head-only divs.
Replace the earlier div-grouping approach with a flat, non-destructive one. Grouping sub-heads into a shared <div> put a <head> after a <p> in ~91% of grouped divs, which is invalid TEI -- the same ill-formed output that shelved section nesting (issues #377, #1074) before. Instead keep the div structure identical to the outline-less output (one div per head) and add a @Level attribute (1 = main section, 2 = sub-section, ...) on heads that can be located in the PDF outline. The outline is the only signal that reveals when a section returns to a higher level, which cannot be inferred from the text: 79% of heads carry no number, so numbering cannot express hierarchy for most documents. A head GROBID misses simply carries no level and affects nothing else; heads absent from the outline are left unmarked. computeSectionLevels matches each SECTION head to the outline (retrying with the section number stripped, since outlines often store the bare title) and sets level = outline depth - minDepth + 1. Removes the grouping helpers (isDescendantOf, numberingContradictsNesting, computeSectionNodes).
…ments) Add the section-head @Level attribute (the outline hierarchy level) to <head>. Also bring the shipped RelaxNG schema in line with what GROBID actually emits: its output has never validated against grobid-home/schemas/rng/Grobid.rng. Add the elements and attributes GROBID produces but the schema omitted: - @coords (PDF bounding boxes) and @lang (bare variant of xml:lang, e.g. on orgName) on att.global, so they are accepted on every element; - <rs> (referencing string, e.g. funders/grants in running text) in model.phrase; - <funder> and <respStmt>/<resp> in model.respLike (title statement); - <listOrg>/<org> in model.listLike, and allowed interleaved with <div>s in <back>; - xsi:schemaLocation on the root <TEI>; - a bare pointer/reference (model.ptrLike) directly under <div> (annex markers); - make @url optional on <graphic> (GROBID emits graphics located only by @coords). Validated with jing over ~275 documents (PMC, a publisher-diverse set, and the dataseer corpus): all validate except a couple carrying malformed @target URIs extracted from corrupt PDFs (a data issue, correctly rejected). The @Level addition is mirrored to the XSD and DTD; the element additions are RelaxNG-only because the shipped XSD has a pre-existing non-deterministic content-model defect that prevents it from loading at all.
lfoppiano
force-pushed
the
feature/use-outline-to-structure-head
branch
from
August 8, 2026 21:31
2bc72ba to
3b22481
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR exploit the PDF outline (when available) to improve the structuring of the sections and sections headers in fulltext. From the text alone you can tell when a new sub-level starts (two heads in a row), but not when a section returns to a higher level. That "ascent" is only knowable from the PDF outline (bookmarks). Section numbering can express it, but on a 1,985-document sample 79% of heads carry no
@nnumber at all and 72% of documents have no numbered head, so numbering cannot carry hierarchy for most papers (Nature/ASM-style "Methods" with unnumbered sub-methods are the typical case).Summary
It uses the outline only to state the level, never to restructure the document. The
<div>structure is unchanged — one<div>per head, byte-identical to the outline-less output — and each head that can be located in the outline gets a@levelattribute (1= main section,2= sub-section, …):level="1|2|3"matches the spelling already used in the fulltext training corpus (grobid-trainer/resources/dataset/fulltext/corpus/tei/), which until now was dead data no parser read.Properties that fall out of this design:
section_titleis scored as the set of head texts, and neither head text nor div structure changes — only an attribute is added.Schema changes
@levelhad to be declared — it was legal only on<title>(the bibliographic sense).While doing so it became clear GROBID's output has never validated against its own shipped
grobid-home/schemas/rng/Grobid.rng. This PR also closes those pre-existing gaps, so the RelaxNG schema now accepts real GROBID output:@coords(PDF bounding boxes)att.global(all elements)@lang(bare variant ofxml:lang, e.g. onorgName)att.global<rs>(referencing string: funders, grant numbers/names in text)model.phrase<funder>,<respStmt>,<resp>model.respLike(title statement)<listOrg>model.listLike, plus interleavable with<div>s in<back>xsi:schemaLocation<TEI>model.ptrLike)<div>(annex markers)@urlmade optional on<graphic>@coords@levelis mirrored to the XSD and DTD. The element additions are RelaxNG-only: the shipped XSD has a pre-existing non-deterministic content-model defect that prevents it from loading at all, so it cannot be validated or meaningfully extended without a separate overhaul.Verification
Five corpora, 15,925 documents, all processed with this build.
@level<head>after<p><head>after<p>is 0 on every document, confirming the output stays structurally valid at scale (this is exactly the failure mode that sank the earlier attempts).jing, samples per corpus): 150/150 on the internal dataset, 96/100 on each HF dataset, 115/116 on a PMC + publisher-diverse set. Every remaining failure is a malformed@targetURI extracted from a corrupt PDF — a data issue the schema correctly rejects, not a schema gap.Also included
PDFALTOOutlineSaxHandler: fixes nested-outline parent tracking. A singlecurrentParentIdwas reset at every</TOCITEMLIST>, so items following a nested list were reattached to the root instead of their real parent. Replaced with a balanced parent-id stack. Java test ported to Kotlin, with nested and malformed outline fixtures.DocumentNode.findNode/findNodeDepth: soft outline-label matching (Ratcliff/Obershelp ≥ 0.90) normalising the non-breaking spaces, soft hyphens and|separators that pdfalto outline labels carry. Heads are matched by their labelled text first, then retried with the section number stripped, since outlines commonly store the bare title ("Results") while the head reads "2. Results".FullTextParser: enables outline extraction for full-text processing.Limitations