Skip to content

Modernization Phase 2.0: modules scaffolding + compile-time baselines #28

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ptesavol merged 2 commits into from
Jul 3, 2026
Merged

Modernization Phase 2.0: modules scaffolding + compile-time baselines #28

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0f2a480
Phase 2.0: modules scaffolding — StreamrModules.cmake, bench.sh, macO…
ptesavol Jul 3, 2026
8c904dd
gitignore the -.json time-trace artifact of CMake compiler detection
ptesavol Jul 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,6 @@ setenvs.sh
*.DS_Store*
packages/streamr-libstreamrproxyclient/dist/android/StreamrProxyClient/libs/arm64-v8a/libstreamrproxyclient.so
packages/streamr-libstreamrproxyclient/dist/android-library-module/StreamrProxyClient/libs/arm64-v8a/libstreamrproxyclient.so
/build-bench-trace/
bench-trace.bin
/-.json
47 changes: 47 additions & 0 deletions MODERNIZATION.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -409,6 +409,53 @@ document/replace in 1.4.
| 2.5 | streamr-trackerless-network, streamr-libstreamrproxyclient | tn `:protos` over NetworkRpc; proxyclient imports only, C header untouched; full iOS XCFramework + Android smoke. **Final metrics** |
| 2.6 | Consolidation (MANDATORY, interleaved) | Per package, once its last dependent is module-based: move declarations into module purview, delete the package's `include/` tree (grep-enforced). End state: no internal headers anywhere; `#include` only for third-party, generated proto, and the C API header. Finalize lint posture; docs |

## Phase 2.0 — Scaffolding + baselines (PR pending)
- `cmake/StreamrModules.cmake` (canonical, synced to all packages): Ninja +
non-AppleClang guards, CMP0155 NEW, `streamr_add_module_library()`
(STATIC + `FILE_SET CXX_MODULES` rooted at `modules/`),
`streamr_enable_imports()` for import-using targets, and the
`STREAMR_IMPORT_STD` opt-in (OFF; requires the CMake-version-specific
experimental UUID to be supplied explicitly). Module scanning stays
globally OFF — targets opt in via the helpers, so clangd's compile
commands stay clean for non-migrated code.
- `bench.sh`: `clean` / `incremental [header]` / `trace` modes measuring the
root single-tree build; prints host/compiler/parallelism with every run.
`trace` uses a separate `build-bench-trace/` tree (shares
`build/vcpkg_installed`) + ClangBuildAnalyzer if installed.

### Baseline (macOS, 2026-07-03)
Apple Silicon dev machine, 10 cores used, Homebrew clang 22.1.8, CMake 4.3,
Debug, Ninja; root single tree (108 TUs: all tests + generated proto +
proxyclient). Single runs on an idle machine.

| Metric | Baseline |
|---|---|
| Clean build | configure 8 s + build **102 s** |
| Incremental: touch `streamr-utils/StreamID.hpp` | 10 TUs, **19 s** |
| Incremental: touch `streamr-logger/SLogger.hpp` | 48 TUs, **62–70 s** |
| Compile CPU split (`-ftime-trace`) | frontend parse **859 s** vs backend codegen **61 s** → **93% parsing** |

ClangBuildAnalyzer expensive headers (cumulative parse, count):
1. `DhtRpc.pb.h` **96.0 s** (33×, avg 2.9 s) — becomes `streamr.dht:protos`
2. `gtest/gtest.h` 84.3 s (87×) — stays `#include` (macros)
3. `SLogger.hpp` **80.8 s** (47×) — becomes a `streamr.logger` partition
4. `Logger.hpp` 66.4 s (50×) — ditto
5. `ConnectionManager.hpp` 38.2 s (4×, **avg 9.5 s per inclusion**)
6. `RpcCommunicator.hpp` 34.7 s (21×)
7. `Identifiers.hpp` 31.2 s (31×)

Top template-instantiation sets (nlohmann parse ~22 s, `std::format` ~13 s,
magic_enum ~13 s) — instantiation cost that modules will NOT move (recorded
so post-migration numbers are read fairly).

The data confirms the plan's premise: >90% of compile CPU is repeated
header parsing, and the top of the expensive list is exactly the
`.pb.h`/streamr-header stack the partitions wrap.

- Linux and iOS baselines: not yet captured (macOS dev iteration is the
primary metric). Capture the Linux numbers on the self-hosted arm64 box
and an iOS build-only timing before the Phase 2.4 checkpoint.

## Lint/IDE survival
- During the façade stage, headers remain the fully-linted source of truth
(`lint.sh` globs only `*.hpp/*.cpp`); `.cppm` added to clang-format only.
Expand Down
115 changes: 115 additions & 0 deletions bench.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
#!/bin/bash

# Compile-time benchmark harness for the C++ modules migration
# (MODERNIZATION.md Part 2). Measures the ROOT single-tree build — the
# workflow the migration optimizes (it compiles every package's tests, the
# generated protobuf sources and the proxy client in one Ninja graph).
# Results are recorded per phase in MODERNIZATION.md.
#
# Usage:
# ./bench.sh clean clean-build wall-clock (root tree only; keeps
# build/vcpkg_installed and the per-package build
# dirs — their exported configs are needed by the
# root configure)
# ./bench.sh incremental [header]
# rebuild after touching a mid-stack header
# (default: streamr-utils/StreamID.hpp, included
# by most dht/trackerless-network TUs)
# ./bench.sh trace clean build in a separate build-bench-trace/
# tree with -ftime-trace; if ClangBuildAnalyzer is
# installed (brew install clang-build-analyzer),
# prints the expensive-headers/templates report
#
# Methodology: run on an otherwise idle machine, 2-3 repetitions, take the
# best. All modes print a single "BENCH <mode>: ..." summary line at the end.

set -e

if [ -z "$VCPKG_ROOT" ]; then
echo "Error: environment not set up. Run 'source install-prerequisities.sh' or 'source setenvs.sh' first."
exit 1
fi

MODE="$1"
BUILD_TYPE="${BENCH_BUILD_TYPE:-Debug}"

print_env() {
echo "bench environment:"
echo " host: $(sysctl -n machdep.cpu.brand_string 2>/dev/null || grep -m1 'model name' /proc/cpuinfo | cut -d: -f2)"
echo " cores: $(sysctl -n hw.ncpu 2>/dev/null || nproc)"
echo " parallelism: ${CMAKE_BUILD_PARALLEL_LEVEL:-ninja default}"
echo " compiler: $(${LLVM_PREFIX:+$LLVM_PREFIX/bin/}clang++ --version | head -1)"
echo " cmake: $(cmake --version | head -1)"
echo " build type: $BUILD_TYPE"
}

configure_and_build() {
# $1 = binary dir, remaining args = extra cmake configure options
local BINDIR="$1"; shift
local T_CONF_START T_CONF_END T_BUILD_END
T_CONF_START=$(date +%s)
cmake -B "$BINDIR" -DCMAKE_BUILD_TYPE=$BUILD_TYPE "$@" .
T_CONF_END=$(date +%s)
cmake --build "$BINDIR"
T_BUILD_END=$(date +%s)
CONFIGURE_SECONDS=$((T_CONF_END - T_CONF_START))
BUILD_SECONDS=$((T_BUILD_END - T_CONF_END))
}

case "$MODE" in
clean)
print_env
# Clean the root tree only: vcpkg_installed stays (dependencies are not
# what is being measured), per-package build dirs stay (the root
# configure resolves sibling packages against their exported configs).
find build -mindepth 1 -maxdepth 1 ! -name vcpkg_installed ! -name .gitignore -exec rm -rf {} +
configure_and_build build
echo ""
echo "BENCH clean: configure ${CONFIGURE_SECONDS}s, build ${BUILD_SECONDS}s (total $((CONFIGURE_SECONDS + BUILD_SECONDS))s)"
;;

incremental)
HEADER="${2:-packages/streamr-utils/include/streamr-utils/StreamID.hpp}"
if [ ! -f "$HEADER" ]; then
echo "Error: header not found: $HEADER"
exit 1
fi
if [ ! -f build/build.ninja ]; then
echo "Error: no configured root build tree; run ./bench.sh clean (or ./install.sh) first."
exit 1
fi
print_env
# Warm the graph so only the header touch is measured.
cmake --build build
touch "$HEADER"
T_START=$(date +%s)
cmake --build build
T_END=$(date +%s)
echo ""
echo "BENCH incremental ($HEADER): $((T_END - T_START))s"
;;

trace)
print_env
# Separate tree: -ftime-trace changes every object file, and the root
# CMakeLists pins VCPKG_INSTALLED_DIR to build/vcpkg_installed, so a
# second binary dir reuses the installed dependencies for free.
rm -rf build-bench-trace
configure_and_build build-bench-trace -DCMAKE_CXX_FLAGS=-ftime-trace
echo ""
echo "BENCH trace: configure ${CONFIGURE_SECONDS}s, build ${BUILD_SECONDS}s (with -ftime-trace)"
if command -v ClangBuildAnalyzer >/dev/null 2>&1; then
ClangBuildAnalyzer --all build-bench-trace bench-trace.bin
ClangBuildAnalyzer --analyze bench-trace.bin
rm -f bench-trace.bin
else
echo "ClangBuildAnalyzer not found (brew install clang-build-analyzer / build from"
echo "https://github.com/aras-p/ClangBuildAnalyzer) — traces are in build-bench-trace/**/*.json"
fi
;;

*)
echo "Usage: ./bench.sh clean | incremental [header] | trace"
exit 1
;;
esac
84 changes: 84 additions & 0 deletions cmake/StreamrModules.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
# CANONICAL COPY — the per-package copies in packages/*/StreamrModules.cmake
# are generated from this file by ./sync-cmake-files.sh. Edit THIS file and
# run the sync script; do not edit the package copies directly.
# (Each package carries its own copy so that it can be published as a
# standalone vcpkg package later.)
#
# C++ named-modules build support (MODERNIZATION.md Part 2). A package
# includes this file when it starts defining module targets. Non-migrated
# packages are unaffected: the root build keeps CMAKE_CXX_SCAN_FOR_MODULES
# OFF globally (clean compile commands for clangd), and the helpers below
# re-enable scanning per target.

# CMake's C++ modules support requires the Ninja generator (Makefiles cannot
# express the dynamically discovered module dependency graph).
if(NOT CMAKE_GENERATOR MATCHES "Ninja")
message(FATAL_ERROR
"C++ modules require the Ninja generator; current generator is "
"'${CMAKE_GENERATOR}'. install-prerequisities.sh/setenvs.sh export "
"CMAKE_GENERATOR=Ninja — source one of them (and ./clean.sh once if "
"the build dir was configured with another generator).")
endif()

# AppleClang has no CMake modules support; the build already uses Homebrew
# LLVM everywhere (homebrewClang.cmake), so this only guards misconfiguration.
if(CMAKE_CXX_COMPILER_ID STREQUAL "AppleClang")
message(FATAL_ERROR
"CMake does not support C++ modules with AppleClang. The build is "
"expected to use Homebrew LLVM (see homebrewClang.cmake / "
"LLVM_PREFIX).")
endif()

# CMP0155 NEW: scan C++20+ sources for module dependencies by default.
# Scanning stays globally disabled via CMAKE_CXX_SCAN_FOR_MODULES OFF until
# a target opts in through the helpers below.
cmake_policy(SET CMP0155 NEW)

# Opt-in `import std;` (experimental in CMake, OFF by default — see the
# MODERNIZATION.md decision: no rollout now). CMake gates the feature behind
# a per-version experimental UUID, so turning this on requires passing the
# UUID for the CMake version in use:
# cmake -DSTREAMR_IMPORT_STD=ON \
# -DCMAKE_EXPERIMENTAL_CXX_IMPORT_STD=<uuid-for-your-cmake> ...
option(STREAMR_IMPORT_STD "Build with experimental 'import std' support" OFF)
if(STREAMR_IMPORT_STD)
if(NOT DEFINED CMAKE_EXPERIMENTAL_CXX_IMPORT_STD)
message(FATAL_ERROR
"STREAMR_IMPORT_STD=ON needs "
"-DCMAKE_EXPERIMENTAL_CXX_IMPORT_STD=<uuid> (the experimental "
"feature UUID documented for your CMake version in "
"Help/dev/experimental.rst of the CMake source).")
endif()
set(CMAKE_CXX_MODULE_STD ON)
endif()

# streamr_add_module_library(<target> FILES <unit.cppm>...)
#
# Defines <target> as a STATIC library whose C++ module interface units are
# the given files (FILE_SET CXX_MODULES, rooted at the package's modules/
# directory). STATIC rather than INTERFACE: module interface units are
# compiled TUs, so even a previously header-only package gains a compiled
# archive when it grows a module.
function(streamr_add_module_library TARGET)
cmake_parse_arguments(ARG "" "" "FILES" ${ARGN})
if(NOT ARG_FILES)
message(FATAL_ERROR "streamr_add_module_library(${TARGET}): FILES is required")
endif()
add_library(${TARGET} STATIC)
target_sources(${TARGET}
PUBLIC
FILE_SET CXX_MODULES
BASE_DIRS ${CMAKE_CURRENT_SOURCE_DIR}/modules
FILES ${ARG_FILES})
set_target_properties(${TARGET} PROPERTIES CXX_SCAN_FOR_MODULES ON)
endfunction()

# streamr_enable_imports(<target>)
#
# Enables module scanning on an existing target whose ordinary .cpp sources
# use `import` (e.g. a test executable of a migrated package). Without this
# the global CMAKE_CXX_SCAN_FOR_MODULES OFF would leave the import edges
# undiscovered and the build would race the BMIs.
function(streamr_enable_imports TARGET)
set_target_properties(${TARGET} PROPERTIES CXX_SCAN_FOR_MODULES ON)
endfunction()
84 changes: 84 additions & 0 deletions packages/streamr-dht/StreamrModules.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
# CANONICAL COPY — the per-package copies in packages/*/StreamrModules.cmake
# are generated from this file by ./sync-cmake-files.sh. Edit THIS file and
# run the sync script; do not edit the package copies directly.
# (Each package carries its own copy so that it can be published as a
# standalone vcpkg package later.)
#
# C++ named-modules build support (MODERNIZATION.md Part 2). A package
# includes this file when it starts defining module targets. Non-migrated
# packages are unaffected: the root build keeps CMAKE_CXX_SCAN_FOR_MODULES
# OFF globally (clean compile commands for clangd), and the helpers below
# re-enable scanning per target.

# CMake's C++ modules support requires the Ninja generator (Makefiles cannot
# express the dynamically discovered module dependency graph).
if(NOT CMAKE_GENERATOR MATCHES "Ninja")
message(FATAL_ERROR
"C++ modules require the Ninja generator; current generator is "
"'${CMAKE_GENERATOR}'. install-prerequisities.sh/setenvs.sh export "
"CMAKE_GENERATOR=Ninja — source one of them (and ./clean.sh once if "
"the build dir was configured with another generator).")
endif()

# AppleClang has no CMake modules support; the build already uses Homebrew
# LLVM everywhere (homebrewClang.cmake), so this only guards misconfiguration.
if(CMAKE_CXX_COMPILER_ID STREQUAL "AppleClang")
message(FATAL_ERROR
"CMake does not support C++ modules with AppleClang. The build is "
"expected to use Homebrew LLVM (see homebrewClang.cmake / "
"LLVM_PREFIX).")
endif()

# CMP0155 NEW: scan C++20+ sources for module dependencies by default.
# Scanning stays globally disabled via CMAKE_CXX_SCAN_FOR_MODULES OFF until
# a target opts in through the helpers below.
cmake_policy(SET CMP0155 NEW)

# Opt-in `import std;` (experimental in CMake, OFF by default — see the
# MODERNIZATION.md decision: no rollout now). CMake gates the feature behind
# a per-version experimental UUID, so turning this on requires passing the
# UUID for the CMake version in use:
# cmake -DSTREAMR_IMPORT_STD=ON \
# -DCMAKE_EXPERIMENTAL_CXX_IMPORT_STD=<uuid-for-your-cmake> ...
option(STREAMR_IMPORT_STD "Build with experimental 'import std' support" OFF)
if(STREAMR_IMPORT_STD)
if(NOT DEFINED CMAKE_EXPERIMENTAL_CXX_IMPORT_STD)
message(FATAL_ERROR
"STREAMR_IMPORT_STD=ON needs "
"-DCMAKE_EXPERIMENTAL_CXX_IMPORT_STD=<uuid> (the experimental "
"feature UUID documented for your CMake version in "
"Help/dev/experimental.rst of the CMake source).")
endif()
set(CMAKE_CXX_MODULE_STD ON)
endif()

# streamr_add_module_library(<target> FILES <unit.cppm>...)
#
# Defines <target> as a STATIC library whose C++ module interface units are
# the given files (FILE_SET CXX_MODULES, rooted at the package's modules/
# directory). STATIC rather than INTERFACE: module interface units are
# compiled TUs, so even a previously header-only package gains a compiled
# archive when it grows a module.
function(streamr_add_module_library TARGET)
cmake_parse_arguments(ARG "" "" "FILES" ${ARGN})
if(NOT ARG_FILES)
message(FATAL_ERROR "streamr_add_module_library(${TARGET}): FILES is required")
endif()
add_library(${TARGET} STATIC)
target_sources(${TARGET}
PUBLIC
FILE_SET CXX_MODULES
BASE_DIRS ${CMAKE_CURRENT_SOURCE_DIR}/modules
FILES ${ARG_FILES})
set_target_properties(${TARGET} PROPERTIES CXX_SCAN_FOR_MODULES ON)
endfunction()

# streamr_enable_imports(<target>)
#
# Enables module scanning on an existing target whose ordinary .cpp sources
# use `import` (e.g. a test executable of a migrated package). Without this
# the global CMAKE_CXX_SCAN_FOR_MODULES OFF would leave the import edges
# undiscovered and the build would race the BMIs.
function(streamr_enable_imports TARGET)
set_target_properties(${TARGET} PROPERTIES CXX_SCAN_FOR_MODULES ON)
endfunction()
Loading
Loading