Handle middle dot (U+00B7) as bullet only outside words - #1524
Conversation
Remove middle dot from the generic bullet_chars pattern and add a dedicated pattern that treats it as a bullet only when not adjacent to word characters. This preserves intra-word uses like Catalan "punt volat" (intel·ligència), chemical formulas (CuSO4·5H2O), and mathematical notation, while still normalizing middle dots used as list markers to the standard bullet character (U+2022).
There was a problem hiding this comment.
Pull request overview
This PR refines Unicode normalization in UnicodeUtil so that U+00B7 (MIDDLE DOT) is only rewritten to the standard bullet (U+2022, •) when it is not between two word characters, preserving legitimate intra-token uses (e.g., Catalan “punt volat”, chemical hydrates).
Changes:
- Removed U+00B7 from the generic
bullet_charsnormalization set. - Added a dedicated
MIDDLE_DOT_BULLET_PATTERNthat only normalizes U+00B7 when it is not between letter/number characters. - Added unit tests covering preservation vs. normalization cases, including a tokenizer regression guard.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| grobid-core/src/main/java/org/grobid/core/utilities/UnicodeUtil.java | Introduces a dedicated middle-dot pattern and applies it after the existing bullet normalization to preserve intra-word dots. |
| grobid-core/src/test/java/org/grobid/core/utilities/UnicodeUtilMiddleDotTest.java | Adds targeted tests for middle-dot behavior and a tokenization regression check. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
Hi @toniher ! Thanks for the PR. Could you please share also a couple of examples of PDF with this issue, so that I can test them? |
|
Hi @lfoppiano ! Thanks for considering it. Any Catalan text would be a good case. I first noticed it here: https://zenodo.org/records/13836801/files/GuiaPublicacio-Zenodo-Article-Catalan-v1.pdf?download=1 , but any other PDF from that community would work, since it is a common situation: https://zenodo.org/communities/recercatala/records . Another one, for instance: https://zenodo.org/records/15261991/files/GCATSSeldem-SCIRE.Article-Catalan-v1.pdf?download=1 I have not managed to find cases for other alike usages, such as hydrates — yet. It appears that authors try to (understandably) avoid mid-dots: https://pubmed.ncbi.nlm.nih.gov/?term=CuSO4%C2%B75H2O |
Remove middle dot from the generic bullet_chars pattern and add a dedicated pattern that treats it as a bullet only when not adjacent to word characters. This preserves intra-word uses like Catalan "punt volat" (intel·ligència), chemical formulas (CuSO4·5H2O), and mathematical notation, while still normalizing middle dots used as list markers to the standard bullet character (U+2022).
Disclaimer: Claude Opus 5.0 support