Accuracy & vRAM: pdfalto vs PyMuPDF vs pdfplumber vsPDFFigures 2.0

[flckv]

Grobid version

9

Operating System and architecture (arm64, amd64, x86, etc.)

No response

What is your Java version

No response

Log and information

I understand that

pdfalto (that grobid9 uses) is different than the models that are popular today that rely on pixel and or semantic layerings, making processing time slow.

However, I noticed there are several technologies other than pdfalto that also use similar methods and so are lightweight.

Heuristic Rules (Programmatic/Low-Level)
Method: Parses the PDF byte stream (code), objects, and font metadata.
Tools: pdfalto, PyMuPDF, pdfplumber, PDFFigures 2.0.
Compute: Lightweight (CPU).
Benchmark Context: Usually evaluated on PubLayNet or DocBank (older standards).

Do you have any testing documentation where we have a better understanding on why pdfalto more accurately with less vRAM extracts tables, figures and formulas than PyMuPDF, pdfplumber or PDFFigures 2.0?

Besides the processing time it seems that even the semantic vision approach can miss parts of images in pdfs e.g. X-PLUG/mPLUG-DocOwl#27 . I wonder if relying on extracting images using pdfalto would be more accurate in unique cases like this?

( I tried your QA chat but couldn't respond me)

Further information

No response

[lfoppiano]

Hi @flckv, thank you for your interest.

pdfalto does indeed not use any visual information, and is only a part of the Grobid architecture, which transforms the pdf document into a XML-based version of it.

There are other alternatives as you write, that are either extracting text or providing programmatic access to the document information: blocks, lines and so on. pdfalto is tightly coupled into grobid and it would not be that easy to replace it. However, it can be improved if there are problems with it as we have access to the source code.

Regarding benchmarks there is one but is limited to a very small amount of documents: https://github.com/lfoppiano/pdf-extraction-benchmarks and recently I've set up a more bench-style, user-friendly version: https://huggingface.co/spaces/sciencialab/pdf-to-text-transformation-bench with the idea, with time, to add more examples especially on large documents or documents with complex reading order.

Adding more example, however would require a accurate manual revision on the ground truth, which is very time consuming.

Pdfalto, however can extract figures, both SVG and bitmap, but, for the former, sometimes there are invisible pieces spread around the paper that makes it a total blow up.

[lfoppiano]

@flckv please check this out: https://huggingface.co/spaces/sciencialab/pdf-to-text-transformation-bench due to lack of time the development is not fast, but I'm planning to extend the small dataset at some point.