[samhocevar]

Using a sorted array with binary search offers the following improvements:

• no more runtime heap allocations to initialize the lookup table
• the data is static and can be directly mapped from rodata
• the compose_pair() function is now about 7 times faster

[yhirose]

@samhocevar, thanks for the patch. I verified the change locally and it looks good to me.

Two things I would like changed before merging.

1. The braces produce 961 warnings per TU on clang

{ (UINT64_C(...) << 42) | ... }, is a braced initializer around a scalar, which clang diagnoses under -Wbraced-scalar-init. That warning is on by default, no -Wall required:

$ c++ -std=c++17 -fsyntax-only -I. anything_including_unicodelib.cpp | grep -c warning:
961

For a header-only library that is a lot of noise for every clang user. GCC does not have this warning, which is why CI stayed green. Removing the braces fixes it (I regenerated and confirmed 0 warnings).

2. Please keep inline const rather than static inline const

static gives the array internal linkage, so every TU that includes the header gets its own 7.7 kB copy in rodata:

this PR : non-external  __ZN7unicodeL26_normalization_compositionE
inline  : weak external __ZN7unicode26_normalization_compositionE

This is the same thing #22 fixed for the other tables. There is no dynamic initialisation here so the cost is lower than before, but the duplication is still there, and inline const keeps the data in rodata just as well.

Combining both points, the generator becomes:

print("inline const uint64_t _normalization_composition[] = {")
for e in sorted((codes[0], codes[1], cp) for cp, codes in items() if not cp in exclusions):
    print("(UINT64_C(0x%08X) << 42) | (UINT64_C(0x%08X) << 21) | 0x%08X," % e)
print("};")

I regenerated the header this way: 0 warnings, symbol is weak external, all tests still pass.

Minor: out-of-range code points can now alias

Because the fields are bit-packed, a second code point above 21 bits spills into the first field:

char32_t out;
unicode::compose_pair(U'A', char32_t(0x300 + (1u << 21)), out);
// before: false
// after : true, out == 0x00C0

It is reachable through to_nfc() too. Only invalid (non-scalar) input triggers this, so it is low severity, but a one-line range check would close it if you think it is worth having.

About the "7 times faster" claim

I could not reproduce that number, and the direction turns out to depend on the access pattern rather than being uniform. compose_pair() in isolation, Apple M-series, Release -O2, median of 5 runs:

pattern	clang / libc++	gcc-15 / libstdc++
small set of pairs hit repeatedly	5.78 -> 22.75 ns (3.9x slower)	8.05 -> 10.36 ns (1.3x slower)
all misses	15.03 -> 12.04 ns (1.25x faster)	20.96 -> 6.64 ns (3.2x faster)
hits scattered over the table	14.22 -> 12.37 ns (1.15x faster)	20.23 -> 15.02 ns (1.35x faster)

Binary search over 961 entries is about 10 dependent loads per lookup and the CPU cannot overlap them, whereas the hash map is 2 loads. So the array only wins once the map's nodes stop fitting in cache, which is what the miss and scattered rows show. Your 7x looks plausible for a miss-heavy microbenchmark on libstdc++, where hashing a std::u32string plus a prime-modulo bucket index is comparatively expensive.

None of this is visible at the API level, though. I benchmarked to_nfc() on both toolchains, with a plain NFD string and with a worst case built from long combining sequences, and the two versions are indistinguishable: run to run variance is around 10 percent, well above any systematic difference. That makes sense since compose_pair() is only reached inside combining sequences and decomposition plus canonical ordering dominate the work.

So no objection on performance grounds. I would just suggest either dropping the 7x from the commit message or stating the environment it was measured in.

Fix the braces and the static and I will merge. Thanks again.

[samhocevar]

Thanks for the through review! Everything makes sense, I will fix all the issues.

I redid the benchmarks with some slightly different test code, with GCC 16, Clang 21.1.8, and whatever Visual Studio 2026 currently ships with. I could not reproduce the ×7 speedup with this specific benchmark (I remember it was obtained with GCC 12). Here is what I found; the data array is filled with random codepoints, but I have tested with ASCII and other distributions, with few noticeable differences except that Clang seems to be slower with very reduced input such as Latin1. Anyway, as you said, the performance of such a small function is not really important when considering the big picture.

#define N 0x8000
extern char32_t data[N];
for (uint32_t n = 0; n < N * N; ++n)
{
    char32_t a = data[n % N], b = data[n * 17 % N], c;
    if (unicode::compose_pair(a, b, c))
        volatile auto d = c;
}

compiler/platform	before	after	speedup
g++ -Os (Linux)	28.52ns	5.70ns	×5.01
g++ -O2 (Linux)	12.45ns	4.43ns	×2.81
clang++ -Os (Linux)	14.30ns	8.87ns	×1.61
clang++ -O2 (Linux)	20.95ns	7.63ns	×2.74
MSVC Release (Windows)	21.16ns	4.93	×4.29

[samhocevar]

Okay, I force-pushed a commit that fixes all issues and remarks (including the aliasing thing, I would feel bad leaving such a bug outstanding).

[yhirose]

Closing this — not because the idea was wrong, but because the table underneath it got rewritten.

Context: 8675d56..160b901 rewrote all of this library's lookup tables, for a downstream user (Culebra) that links a static binary per program and only wants the tables a program actually uses. Before: calling .lower() alone pulled in 1.5 MB. After: 213 KB.

On #23's two goals:

• No more heap allocations, static rodata array: already done in 9603809, before this PR existed.
• compose_pair() faster: this part was still worth doing, so I rebased it onto the new tables and merged it as 160b901, keeping your authorship. Packing (first, second, composed) into one uint64_t instead of a 12-byte struct: 961 entries, 4,096 fewer bytes linked, and to_nfc() over 200k characters went from 18.6 ms to 17.6 ms. (That's the whole to_nfc() call, not compose_pair() alone, so it's not directly comparable to the 7x you measured.)

Thank you for your fine contribution!