Skip to content

fix: isExactMatch: strip diacritics for accent-aware exact matching - #2258

Open
YannBirba wants to merge 1 commit into
futo-org:masterfrom
YannBirba:fix/exact-match-strip-diacritics
Open

fix: isExactMatch: strip diacritics for accent-aware exact matching#2258
YannBirba wants to merge 1 commit into
futo-org:masterfrom
YannBirba:fix/exact-match-strip-diacritics

Conversation

@YannBirba

Copy link
Copy Markdown

This PR was created by human (me), but all the code LLM generated.
I hope it's okay for you.

Problem

When the GGML language model suggests an accented word (e.g. Garçon) for a
user who typed an unaccented ASCII input (garcon), the exact-match scoring
rule at org_futo_inputmethod_latin_xlm_LanguageModel.cpp:1273-1286 fires
incorrectly:

// Exact match rule
bool hasExactMatch = false;
for(const auto &result : results) {
    if(isExactMatch(result.second, partialWordString)) {
        hasExactMatch = true;
    }
}
if(hasExactMatch){
    for(auto &result : results) {
        if(!isExactMatch(result.second, partialWordString)) {
            result.first -= 1.0f;  // <-- penalizes accented candidates
        }
    }
}

isExactMatch previously only lowercased and stripped ', -, and spaces.
UTF-8 accented characters (é, ç, ê, …) were compared byte-for-byte,
so "Garçon" never matched "garcon". The -1.0 penalty pushed accented
candidates below the unaccented form, defeating the LM's accent predictions.

Root cause

isExactMatch did not normalize Unicode diacritics to their ASCII base forms
before comparison. A user typing garcon on an ASCII keyboard would always
see Garcon instead of Garçon, even when the LM correctly predicted the
accented form.

Fix

Add append_diacritic_normalized(), a lightweight UTF-8 normalizer that maps
accented characters to their base ASCII equivalents:

  • Latin-1 Supplement (U+00C0-U+00FF, 0xC3 prefix): all common
    European accents (French, Spanish, Italian, Portuguese, German, Nordic)
    — including uppercase variants (À-Ÿ) and ligatures (Æ→"ae").
  • Latin Extended-A: Œ/œ (0xC5 0x92/0x93) → "oe".

Unmapped sequences (Ð, Þ, ß, …) keep their original bytes, preserving
stock matching behavior.

Testing

Added native/jni/tests/test_isexactmatch.cpp — a standalone test
(compilable with g++ -std=c++17) covering:

Category Examples
French lowercase accents garçon↔garcon, café↔cafe, être↔etre
Mixed case Garçon↔GARCON, CAFÉ↔CAFE
Ligatures cœur↔coeur, Œuvre↔Oeuvre
Separators stripped l'homme↔lhomme, Hello-World↔helloworld
Negative cases maison≠maisonn, chat≠chien
Unmapped sequences ðtest≠dtest (preserves stock behavior)
33/33 tests passed

Also tested on-device (Android emulator, AZERTY layout) with the custom
GGML model:

Typed Expected Result
garcon ␣ Garçon
Xafe ␣ Café
bonjor ␣ Bonjour
etre ␣ Être
francais ␣ Français

Scope & limitations

  • Covers Latin-1 Supplement + Œ/œ. Other scripts (Cyrillic, Arabic, CJK)
    are untouched and preserve stock byte-for-byte comparison.
  • ñ normalizes to n for exact-match purposes (intentional: beam search
    should not penalize ñ when user typed n).
  • Trade-off: words differing only by accent (a/à, ou/où) will both count
    as exact matches. The LM probability still decides ranking; auto-correction
    thresholds guard unintended replacements.

The isExactMatch function compares LM output against the typed word byte-
for-byte after lowercasing. Accented UTF-8 characters (é, ç, ê, …) never
matched their ASCII counterparts, causing the -1.0 exact-match penalty to
push accented candidates below the unaccented form.

Add append_diacritic_normalized() which maps Latin-1 Supplement and
Latin Extended-A accented characters to their ASCII base forms before
comparison. Unmapped sequences preserve stock behavior.

Also add native/jni/tests/test_isexactmatch.cpp with 33 tests covering
French accents, ligatures, case, separators, and edge cases.

Assisted-by: Big Pickle <noreply@opencode.ai>
@futo-cla-pr-labler

Copy link
Copy Markdown

Please sign our contributor license agreement at https://cla.futo.tech

@futo-cla-pr-labler

Copy link
Copy Markdown

Please sign our contributor license agreement at https://cla.futo.tech

@YannBirba

Copy link
Copy Markdown
Author

Just signed the CLA

@Zvonimir-FUTO Zvonimir-FUTO added the Bug Something isn't working label Aug 27, 2026
@Zvonimir-FUTO Zvonimir-FUTO added CLA-not-signed and removed Bug Something isn't working CLA-signed labels Aug 27, 2026
@Zvonimir-FUTO
Zvonimir-FUTO requested a review from abb128 August 27, 2026 10:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants