fix: isExactMatch: strip diacritics for accent-aware exact matching - #2258
Open
YannBirba wants to merge 1 commit into
Open
fix: isExactMatch: strip diacritics for accent-aware exact matching#2258YannBirba wants to merge 1 commit into
YannBirba wants to merge 1 commit into
Conversation
The isExactMatch function compares LM output against the typed word byte- for-byte after lowercasing. Accented UTF-8 characters (é, ç, ê, …) never matched their ASCII counterparts, causing the -1.0 exact-match penalty to push accented candidates below the unaccented form. Add append_diacritic_normalized() which maps Latin-1 Supplement and Latin Extended-A accented characters to their ASCII base forms before comparison. Unmapped sequences preserve stock behavior. Also add native/jni/tests/test_isexactmatch.cpp with 33 tests covering French accents, ligatures, case, separators, and edge cases. Assisted-by: Big Pickle <noreply@opencode.ai>
|
Please sign our contributor license agreement at https://cla.futo.tech |
|
Please sign our contributor license agreement at https://cla.futo.tech |
Author
|
Just signed the CLA |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR was created by human (me), but all the code LLM generated.
I hope it's okay for you.
Problem
When the GGML language model suggests an accented word (e.g.
Garçon) for auser who typed an unaccented ASCII input (
garcon), the exact-match scoringrule at
org_futo_inputmethod_latin_xlm_LanguageModel.cpp:1273-1286firesincorrectly:
isExactMatchpreviously only lowercased and stripped',-, and spaces.UTF-8 accented characters (
é,ç,ê, …) were compared byte-for-byte,so
"Garçon"never matched"garcon". The -1.0 penalty pushed accentedcandidates below the unaccented form, defeating the LM's accent predictions.
Root cause
isExactMatchdid not normalize Unicode diacritics to their ASCII base formsbefore comparison. A user typing
garconon an ASCII keyboard would alwayssee
Garconinstead ofGarçon, even when the LM correctly predicted theaccented form.
Fix
Add
append_diacritic_normalized(), a lightweight UTF-8 normalizer that mapsaccented characters to their base ASCII equivalents:
0xC3prefix): all commonEuropean accents (French, Spanish, Italian, Portuguese, German, Nordic)
— including uppercase variants (À-Ÿ) and ligatures (Æ→"ae").
0xC5 0x92/0x93) → "oe".Unmapped sequences (Ð, Þ, ß, …) keep their original bytes, preserving
stock matching behavior.
Testing
Added
native/jni/tests/test_isexactmatch.cpp— a standalone test(compilable with
g++ -std=c++17) covering:Also tested on-device (Android emulator, AZERTY layout) with the custom
GGML model:
Scope & limitations
are untouched and preserve stock byte-for-byte comparison.
ñnormalizes tonfor exact-match purposes (intentional: beam searchshould not penalize
ñwhen user typedn).as exact matches. The LM probability still decides ranking; auto-correction
thresholds guard unintended replacements.