Modernization Phase 2.6: Part 2 closeout — consolidation memo, NDK r29 modules experiment [androidbuild]

.github/workflows/validateandroid.yml

@@ -24,6 +24,20 @@ jobs:
           fetch-depth: 0
           submodules: true
           token: ${{ secrets.GITHUB_TOKEN }}
+      # The runner's default ANDROID_NDK is an older LTS (r27). Use the
+      # latest stable NDK preinstalled on the image (r29 = clang 21).
+      # Modules build on r27+ too (StreamrModules.cmake floors at NDK
+      # clang 18) — building with the latest stable is hygiene, not a
+      # modules requirement.
+      - name: use latest stable NDK
+        run: |
+          echo "ANDROID_NDK=$ANDROID_NDK_LATEST_HOME" >> "$GITHUB_ENV"
+          echo "ANDROID_NDK_HOME=$ANDROID_NDK_LATEST_HOME" >> "$GITHUB_ENV"
+          echo "ANDROID_NDK_ROOT=$ANDROID_NDK_LATEST_HOME" >> "$GITHUB_ENV"
+          # NB: the prebuilt dir is named darwin-x86_64 even on ARM Macs
+          # (universal binaries under the legacy name).
+          "$ANDROID_NDK_LATEST_HOME"/toolchains/llvm/prebuilt/darwin-*/bin/clang --version | head -1
+        shell: bash
       - name: install
         uses: ./.github/workflows/reusable/cached-install
       # No lint step: linting is host-independent and runs on the

CMakeLists.txt

@@ -7,10 +7,9 @@ set(VCPKG_INSTALLED_DIR ${CMAKE_CURRENT_SOURCE_DIR}/build/vcpkg_installed)
 set (VCPKG_OVERLAY_TRIPLETS ENV{VCPKG_OVERLAY_TRIPLETS})
 set(CMAKE_CXX_STANDARD 26)
 set(CMAKE_CXX_STANDARD_REQUIRED ON)
-# No C++ modules in the codebase yet: CMake >= 3.28 (CMP0155) would otherwise
-# scan all C++20+ sources for module imports, injecting GCC scanning flags
-# (-fmodules-ts, -fmodule-mapper, -fdeps-format) into compile_commands.json,
-# which clangd rejects. The modules migration will turn scanning back on.
+# Module scanning stays OFF globally so that non-module TUs keep clean
+# compile commands for clangd; targets with module units or import-using
+# sources opt back in via the StreamrModules.cmake helpers.
 set(CMAKE_CXX_SCAN_FOR_MODULES OFF)
 include(cmake/homebrewClang.cmake)
 

MODERNIZATION.md

@@ -632,7 +632,7 @@ before any consolidation gains.
   trackerless-network + proxyclient downstream builds unchanged; root
   tree 307/307; full lint green.
 
-## Phase 2.5 — trackerless-network + proxyclient, FINAL METRICS (PR pending)
+## Phase 2.5 — trackerless-network + proxyclient, FINAL METRICS ✅ (PR #33, merged)
 The migration surface is complete: every C++ package that exports an API
 now ships a module.
 - **streamr-trackerless-network**: `:protos` over the NetworkRpc trio
@@ -651,13 +651,35 @@ now ships a module.
 - New standing gate applied (2.4 lesson): a LOCAL RELEASE build+test of
   the migrated package (caught nothing this time — the folly/protobuf
   overlay patches hold).
+- **The Android smoke gate (first Android build since the 1.3 dependency
+  wave) earned its keep twice**:
+  1. Upstream's 2026 folly port silently dropped Android from its
+     `supports` expression → `./install.sh --android` failed outright at
+     dependency install. The folly overlay re-allows Android
+     (JUSTIFICATION.md updated); **folly 2026 builds fine on the NDK**.
+  2. **`import streamr.json` failed on Android** — every imported name
+     reported as undeclared (the `toJson` overload set etc.). Diagnosed
+     AT THE TIME as NDK-clang `export using` overload bugs; resolved by
+     forcing `STREAMR_MODULES_SUPPORTED=OFF` on Android (libraries built
+     textually from headers, module units skipped via the STATIC +
+     stub-source fallback so PUBLIC usage requirements stay identical,
+     import-using test/example targets not built). The Android workflow
+     also stopped running lint (host-independent; covered by
+     validate.yml; import-using test files have no compile commands on
+     an Android tree).
+     CORRECTED in 2.6: the compiler was never the problem — the real
+     cause was a `-pthread` BMI configuration mismatch (see the 2.6
+     memo). Android now builds the modules on NDK r27+ (clang >= 18);
+     the textual fallback remains only for older NDKs.
 
 ### FINAL FAÇADE-STAGE METRICS (vs Phase 2.0 baselines, macOS, idle)
 | Metric | Baseline | Final (7 packages migrated) | Target | Verdict |
 |---|---|---|---|---|
 | Clean root build | 102 s | **78 s (−24%)** | −25% | ✅ effectively met (78–88 s across repeats) |
 | `SLogger.hpp` touch | 62–70 s | 58 s (−6…−12%) | −40% | ❌ not yet — consolidation work |
 | `StreamID.hpp` touch | 19 s | 32 s (+68%) | −40% | ❌ regression — BMI-chain latency |
+| Expensive headers (ClangBuildAnalyzer) | `DhtRpc.pb.h` #1 (96 s, 33×); SLogger/Logger top-4 | `DhtRpc.pb.h` #9 (15.7 s, 6×); SLogger/Logger GONE from list; #1 = `gtest.h` (stays `#include` by design) | pb.h/folly leave the top | ✅ met |
+| Total frontend parse CPU | 859 s | 723 s (−16%, incl. the module units) | — | ✅ |
 
 **Honest reading**: the clean-build target is effectively met already at
 the façade stage (test TUs load BMIs instead of re-parsing header
@@ -677,6 +699,64 @@ sequencing; the numbers now quantify exactly why consolidation matters.
   exclude import-using files from clangd-tidy until clangd modules support
   stabilizes.
 
+## Phase 2.6 — Consolidation: DECISION MEMO (owner decision required)
+The planned end state (user-confirmed 2026-07): NO internal headers —
+all code in `.cppm` module units, `#include` only for third-party,
+generated proto, and the C API header. Consolidation is what delivers the
+so-far-unmet incremental-rebuild targets (a partition edit stops
+invalidating whole-package BMIs).
+
+**The 2.5 Android blocker is RESOLVED (2.6).** The Phase 2.5 failure
+(`import streamr.json` → every imported name "undeclared") was NOT an
+NDK-clang capability gap. Root cause, found by reproducing the CI
+failure locally and bisecting against NDK r27/r28/r29: the import-using
+test TUs compile with `-pthread` (inherited from GTest's
+`Threads::Threads`) while the module libraries' BMIs were compiled
+without it; on Android `-pthread` toggles a language option that clang
+validates when loading a module file
+(`-Wmodule-file-config-mismatch` — the FIRST error in the log; the
+"undeclared identifier" flood is its cascade), so the BMI silently
+fails to load. The raw NDK compilers (clang 18, 19 and 21) all compile
+and import the façade modules correctly when invoked by hand. Fix: the
+module-library helpers in StreamrModules.cmake make `-pthread` a PUBLIC
+compile option on Android, so the BMI and every importer always agree.
+Android CI keeps using the runner's latest stable NDK (r29, clang 21);
+`STREAMR_MODULES_SUPPORTED` keeps a version floor at the oldest
+verified NDK clang (18 = r27), with the textual fallback below it.
+
+### Preconditions for starting consolidation
+1. ~~A modules-capable NDK on Android~~ **RESOLVED (2.6)** — the 2.5
+   failure was the `-pthread` BMI mismatch above, not the compiler.
+   Android builds the façade modules on NDK r27+ (verified locally on
+   r27 and r29 for streamr-json + streamr-eventemitter, and on the
+   Phase 2.6 PR's androidbuild leg for the full monorepo).
+2. clangd modules support matures enough to lint purview code (today it
+   mis-unifies preamble/BMI types; with code in partitions there is no
+   header fallback for the linter — coverage would regress from "full"
+   to "none" for consolidated code).
+3. The dht intra-package include cycles (connection/endpoint cluster)
+   must become a partition DAG — untangle or use coarse per-cluster
+   partitions (deferred from 2.4 by design).
+
+### What consolidation buys (quantified at the 2.4/2.5 checkpoints)
+- The −40% incremental-rebuild targets (currently: SLogger touch −6…−12%,
+  StreamID touch REGRESSED +68% from BMI-chain invalidation).
+- The final IWYU/ODR hygiene of the no-headers end state.
+
+### Recommended path (when unblocked)
+Leaf-first and interleaved as originally planned — eventemitter as the
+consolidation canary, one package per PR, `bench.sh` measured at every
+step (the 2.4 lesson: measure before generalizing), headers deleted
+per-package with a grep-enforced "nothing includes them" gate.
+
+### Interim posture (adopted now)
+The façade stage is COMPLETE and delivers: uniform `import streamr.<pkg>`
+consumption, −24% clean builds, 250+ tests through import, and module
+infrastructure exercised on macOS/Linux/iOS/Android. Headers remain the
+linted source of truth. With the Android blocker resolved, consolidation
+is gated only by preconditions 2 (clangd lint coverage of purview code)
+and 3 (dht partition DAG). This is a stable resting point.
+
 ## Success metrics (measured macOS + Linux, end of Phase 2.5)
 - ≥25% clean-build wall-clock reduction (dev build with tests).
 - ≥40% reduction on incremental rebuild after touching a mid-stack header

README.md

@@ -63,6 +63,44 @@ The codebase is built as C++26 (`CMAKE_CXX_STANDARD 26`, required) everywhere, b
 
 Practical rule: header-only features (ranges, `contains`, concepts, formatting of standard types, etc.) work everywhere; features needing runtime-library symbols (e.g. new locale facets, some `std::print`/filesystem edges) are gated by the iOS deployment target — the iOS build will tell you.
 
+#### C++ named modules
+
+Every API-exporting package ships a C++ named module (`streamr.json`,
+`streamr.logger`, `streamr.eventemitter`, `streamr.utils`,
+`streamr.protorpc`, `streamr.dht`, `streamr.trackerlessnetwork`). The
+package test suites consume them with a single `import streamr.<pkg>;`.
+
+The modules are currently *façades*: each package's `modules/` directory
+holds a coarse `:all` interface unit that includes the public headers in
+its global module fragment and re-exports the public names (plus a
+`:protos` unit over generated protobuf code where applicable). The
+headers remain the source of truth — `#include` and `import` consumers
+can be mixed freely.
+
+Rules of thumb when writing import-using code:
+
+* Include what you use: `import` does not leak transitive std/third-party
+  headers the way textual inclusion did.
+* Third-party libraries (folly, boost, gtest, protobuf) stay `#include`.
+* Do **not** also textually include monorepo headers that reach the same
+  third-party stacks in an import-using file — import the sibling module
+  instead (clangd's experimental modules support mis-diagnoses the mix).
+* A target whose sources `import` a module must link that module's target
+  directly (`streamr::streamr-<pkg>`) and call
+  `streamr_enable_imports(<target>)` — see `cmake/StreamrModules.cmake`.
+* **Android builds the modules with NDK r27+** (clang 18; the CI uses
+  the latest stable NDK, currently r29). With an older NDK the
+  libraries build textually from the headers and test targets are
+  skipped — handled automatically by `STREAMR_MODULES_SUPPORTED` in
+  `StreamrModules.cmake`. Note for module authors: the module-library
+  helpers make `-pthread` part of the PUBLIC interface on Android —
+  clang refuses to load a BMI whose POSIX-thread setting differs from
+  the importing TU's.
+
+The plan for consolidating code out of headers into the module units
+(and the platform preconditions for it) is in
+[MODERNIZATION.md](MODERNIZATION.md).
+
 ### Install all the dependencies and build the SDK for MacOS and Linux
 
 ```bash

cmake/StreamrModules.cmake

@@ -10,16 +10,23 @@
 # OFF globally (clean compile commands for clangd), and the helpers below
 # re-enable scanning per target.
 
-# Android builds use the NDK's clang (18/19), whose C++ modules support is
-# too immature for the façade modules (known `export using` overload-set
-# bugs — observed: `import streamr.json;` fails to provide the toJson
-# overloads). Android consumes the ordinary headers instead (they remain the
-# source of truth during the façade stage), the module units are skipped,
-# and packages must not build their import-using test targets
-# (STREAMR_MODULES_SUPPORTED below guards them). Revisit when the NDK ships
-# clang >= 22 — consolidation (2.6) requires it, as headers disappear then.
+# Android builds use the NDK's clang. The façade modules build correctly
+# with NDK r27+ (clang 18) — the failure previously attributed to compiler
+# immaturity was a -pthread BMI configuration mismatch, fixed in the
+# helpers below. Keep a version floor at the oldest verified NDK clang
+# (18 = r27): with an older NDK, Android consumes the ordinary headers
+# instead (they remain the source of truth during the façade stage), the
+# module units are skipped, and import-using test/example targets are not
+# built (STREAMR_MODULES_SUPPORTED guards them).
 if(VCPKG_TARGET_TRIPLET MATCHES "android" OR CMAKE_SYSTEM_NAME STREQUAL "Android")
-    set(STREAMR_MODULES_SUPPORTED OFF)
+    if(CMAKE_CXX_COMPILER_VERSION VERSION_GREATER_EQUAL 18)
+        set(STREAMR_MODULES_SUPPORTED ON)
+    else()
+        set(STREAMR_MODULES_SUPPORTED OFF)
+        message(STATUS
+            "C++ modules disabled: NDK clang ${CMAKE_CXX_COMPILER_VERSION} "
+            "< 18 (use NDK r27+ for modules on Android)")
+    endif()
 else()
     set(STREAMR_MODULES_SUPPORTED ON)
 endif()
@@ -104,6 +111,15 @@ function(streamr_add_module_library TARGET)
     # BMI-compiling target on the consumer side and requires the standard
     # level to be part of the exported usage requirements.
     target_compile_features(${TARGET} PUBLIC cxx_std_26)
+    # Importers must agree with the BMI on POSIX-thread support: clang
+    # validates language options when loading a module file. On Android,
+    # -pthread is not implied — test executables inherit it from GTest's
+    # Threads::Threads while library targets do not, so a BMI built
+    # without it cannot be loaded by an import-using test TU (observed:
+    # -Wmodule-file-config-mismatch, then every imported name reported as
+    # undeclared). Making the flag part of the PUBLIC interface keeps the
+    # BMI and every importer consistent. No-op on other platforms.
+    target_compile_options(${TARGET} PUBLIC $<$<PLATFORM_ID:Android>:-pthread>)
 endfunction()
 
 # streamr_target_module_sources(<target> FILES <unit.cppm>...)
@@ -127,6 +143,8 @@ function(streamr_target_module_sources TARGET)
         FILES ${ARG_FILES})
     set_target_properties(${TARGET} PROPERTIES CXX_SCAN_FOR_MODULES ON)
     target_compile_features(${TARGET} PUBLIC cxx_std_26)
+    # See streamr_add_module_library: BMI/importer -pthread consistency.
+    target_compile_options(${TARGET} PUBLIC $<$<PLATFORM_ID:Android>:-pthread>)
 endfunction()
 
 # streamr_enable_imports(<target>)

packages/streamr-dht/CMakeLists.txt

@@ -5,10 +5,9 @@ include(homebrewClang.cmake)
 
 set(CMAKE_CXX_STANDARD 26)
 set(CMAKE_CXX_STANDARD_REQUIRED ON)
-# No C++ modules in the codebase yet: CMake >= 3.28 (CMP0155) would otherwise
-# scan all C++20+ sources for module imports, injecting GCC scanning flags
-# (-fmodules-ts, -fmodule-mapper, -fdeps-format) into compile_commands.json,
-# which clangd rejects. The modules migration will turn scanning back on.
+# Module scanning stays OFF globally so that non-module TUs keep clean
+# compile commands for clangd; targets with module units or import-using
+# sources opt back in via the StreamrModules.cmake helpers.
 set(CMAKE_CXX_SCAN_FOR_MODULES OFF)
 set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
 

packages/streamr-dht/StreamrModules.cmake

@@ -10,16 +10,23 @@
 # OFF globally (clean compile commands for clangd), and the helpers below
 # re-enable scanning per target.
 
-# Android builds use the NDK's clang (18/19), whose C++ modules support is
-# too immature for the façade modules (known `export using` overload-set
-# bugs — observed: `import streamr.json;` fails to provide the toJson
-# overloads). Android consumes the ordinary headers instead (they remain the
-# source of truth during the façade stage), the module units are skipped,
-# and packages must not build their import-using test targets
-# (STREAMR_MODULES_SUPPORTED below guards them). Revisit when the NDK ships
-# clang >= 22 — consolidation (2.6) requires it, as headers disappear then.
+# Android builds use the NDK's clang. The façade modules build correctly
+# with NDK r27+ (clang 18) — the failure previously attributed to compiler
+# immaturity was a -pthread BMI configuration mismatch, fixed in the
+# helpers below. Keep a version floor at the oldest verified NDK clang
+# (18 = r27): with an older NDK, Android consumes the ordinary headers
+# instead (they remain the source of truth during the façade stage), the
+# module units are skipped, and import-using test/example targets are not
+# built (STREAMR_MODULES_SUPPORTED guards them).
 if(VCPKG_TARGET_TRIPLET MATCHES "android" OR CMAKE_SYSTEM_NAME STREQUAL "Android")
-    set(STREAMR_MODULES_SUPPORTED OFF)
+    if(CMAKE_CXX_COMPILER_VERSION VERSION_GREATER_EQUAL 18)
+        set(STREAMR_MODULES_SUPPORTED ON)
+    else()
+        set(STREAMR_MODULES_SUPPORTED OFF)
+        message(STATUS
+            "C++ modules disabled: NDK clang ${CMAKE_CXX_COMPILER_VERSION} "
+            "< 18 (use NDK r27+ for modules on Android)")
+    endif()
 else()
     set(STREAMR_MODULES_SUPPORTED ON)
 endif()
@@ -104,6 +111,15 @@ function(streamr_add_module_library TARGET)
     # BMI-compiling target on the consumer side and requires the standard
     # level to be part of the exported usage requirements.
     target_compile_features(${TARGET} PUBLIC cxx_std_26)
+    # Importers must agree with the BMI on POSIX-thread support: clang
+    # validates language options when loading a module file. On Android,
+    # -pthread is not implied — test executables inherit it from GTest's
+    # Threads::Threads while library targets do not, so a BMI built
+    # without it cannot be loaded by an import-using test TU (observed:
+    # -Wmodule-file-config-mismatch, then every imported name reported as
+    # undeclared). Making the flag part of the PUBLIC interface keeps the
+    # BMI and every importer consistent. No-op on other platforms.
+    target_compile_options(${TARGET} PUBLIC $<$<PLATFORM_ID:Android>:-pthread>)
 endfunction()
 
 # streamr_target_module_sources(<target> FILES <unit.cppm>...)
@@ -127,6 +143,8 @@ function(streamr_target_module_sources TARGET)
         FILES ${ARG_FILES})
     set_target_properties(${TARGET} PROPERTIES CXX_SCAN_FOR_MODULES ON)
     target_compile_features(${TARGET} PUBLIC cxx_std_26)
+    # See streamr_add_module_library: BMI/importer -pthread consistency.
+    target_compile_options(${TARGET} PUBLIC $<$<PLATFORM_ID:Android>:-pthread>)
 endfunction()
 
 # streamr_enable_imports(<target>)

packages/streamr-eventemitter/CMakeLists.txt

@@ -3,10 +3,9 @@ set(CMAKE_POLICY_DEFAULT_CMP0077 NEW)
 
 set(CMAKE_CXX_STANDARD 26)
 set(CMAKE_CXX_STANDARD_REQUIRED ON)
-# No C++ modules in the codebase yet: CMake >= 3.28 (CMP0155) would otherwise
-# scan all C++20+ sources for module imports, injecting GCC scanning flags
-# (-fmodules-ts, -fmodule-mapper, -fdeps-format) into compile_commands.json,
-# which clangd rejects. The modules migration will turn scanning back on.
+# Module scanning stays OFF globally so that non-module TUs keep clean
+# compile commands for clangd; targets with module units or import-using
+# sources opt back in via the StreamrModules.cmake helpers.
 set(CMAKE_CXX_SCAN_FOR_MODULES OFF)
 include(homebrewClang.cmake)