Skip to content

perf(hybrid): tune approved remote expert fanout per layer #181

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
JustVugg merged 3 commits into from
Oct 1, 2026
Merged

perf(hybrid): tune approved remote expert fanout per layer #181

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8664921
Tune approved Hybrid expert fanout using per-layer measurements
JustVugg Sep 25, 2026
86de3ac
perf(hybrid): pace costly probes and verify borderline resident results
JustVugg Oct 1, 2026
e7d5728
fix(hybrid): account for work below clock resolution
JustVugg Oct 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,8 +120,10 @@ jobs:
- name: Resident Segment Hybrid gate
working-directory: lumabri
run: |
make test-segment-hybrid test_home_expert ENGINE=../colibri/c
make test-segment-hybrid test_home_expert build/bench_home_hybrid ENGINE=../colibri/c
./test_home_expert
./build/bench_home_hybrid --numeric-reference-selftest
./build/bench_home_hybrid --numeric-capture-selftest "$(mktemp -d)"
- name: Segment failover gate (fresh fencing, checkpoint replay)
working-directory: colibri/c
run: |
Expand Down Expand Up @@ -320,6 +322,7 @@ jobs:
python3 tools/package_household.py --colibri-root ../colibri --output 'build/native candidate'
python3 tests/integration/home_flow_test.py --runtime-dir 'build/native candidate/bin' --models-dir build/home-flow-models --resident-default --expect-metrics --stall-preparation-ui
python3 tests/integration/home_flow_test.py --runtime-dir 'build/native candidate/bin' --models-dir build/home-flow-models --resident-default --expect-hybrid --expect-metrics
python3 tests/integration/home_flow_test.py --runtime-dir 'build/native candidate/bin' --models-dir build/home-flow-models --resident-default --expect-hybrid --expect-hybrid-fanout --expect-metrics
python3 tests/integration/home_flow_test.py --runtime-dir 'build/native candidate/bin' --models-dir build/home-flow-models --repeat-cached --clear-cached --expect-metrics --expect-calibration
python3 tests/integration/home_flow_test.py --runtime-dir 'build/native candidate/bin' --models-dir build/home-flow-models --kill-donor
python3 tests/integration/home_flow_test.py --runtime-dir 'build/native candidate/bin' --models-dir build/home-flow-models --expect-disjoint-plans --context 256
Expand Down Expand Up @@ -505,6 +508,7 @@ jobs:
fi
python3 tests/integration/home_flow_test.py --runtime-dir 'build/native candidate/bin' --models-dir build/home-flow-models --resident-default --expect-metrics
python3 tests/integration/home_flow_test.py --runtime-dir 'build/native candidate/bin' --models-dir build/home-flow-models --resident-default --expect-hybrid --expect-metrics
python3 tests/integration/home_flow_test.py --runtime-dir 'build/native candidate/bin' --models-dir build/home-flow-models --resident-default --expect-hybrid --expect-hybrid-fanout --expect-metrics
python3 tests/integration/home_flow_test.py --runtime-dir 'build/native candidate/bin' --models-dir build/home-flow-models --repeat-cached --clear-cached --expect-metrics --expect-calibration
python3 tests/integration/home_flow_test.py --runtime-dir 'build/native candidate/bin' --models-dir build/home-flow-models --kill-donor
python3 tests/integration/home_flow_test.py --runtime-dir 'build/native candidate/bin' --models-dir build/home-flow-models --expect-disjoint-plans --context 256
Expand Down
12 changes: 7 additions & 5 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ lumabri test_chat_ui: src/runtime/lumabri_weight_cache.h
lumabri test_chat_ui: src/runtime/lumabri_prepare_progress.h
lumabri test_chat_ui test_memory_budget test_cluster segment_node test_calibration $(SHIM_LIB): src/runtime/lumabri_prepare_limits.h
lumabri test_chat_ui test_home test_hybrid_parallel test_accum_order test_local_fallback test_verify_failover test_nat_adopt: lumabri_home_hybrid.h
test_hybrid_parallel test_accum_order test_local_fallback test_verify_failover test_nat_adopt build/segment_hybrid_bridge.o: src/runtime/lumabri_hybrid_policy.h
test_hybrid_parallel test_accum_order test_local_fallback test_verify_failover test_nat_adopt build/segment_hybrid_bridge.o build/bench_home_hybrid: src/runtime/lumabri_hybrid_policy.h

test_weight_cache: tests/c/test_weight_cache.c src/runtime/lumabri_weight_cache.h
$(CC) $(CPPFLAGS) $(CFLAGS) tests/c/test_weight_cache.c -o $@
Expand Down Expand Up @@ -61,7 +61,7 @@ endif
# regressions would make this gate depend on whichever checkout ENGINE names.
check-warnings:
$(MAKE) -B all test_relay_exec test_swarm_fed test_key_rotation \
test_hedge test_local_fallback test_nat_adopt test_verify_failover test_rtt_refresh test_segment_v2 test_exec2 test_accum_order test_residency_report test_model_family test_planner test_cluster test_memory_budget test_calibration test_metrics \
test_hedge test_local_fallback test_nat_adopt test_verify_failover test_rtt_refresh test_segment_v2 test_exec2 test_accum_order test_hybrid_parallel test_residency_report test_model_family test_planner test_cluster test_memory_budget test_calibration test_metrics \
test_segment_discovery test_swarm_detail test_relay_rate test_machine \
test_meminfo test_compute_lease test_content_filter \
test_scheduler test_run_gate test_inventory test_home test_chat_ui test_weight_cache test_planner_io \
Expand Down Expand Up @@ -484,7 +484,7 @@ test_tcp_latency: tests/c/test_tcp_latency.c $(SECURE_DEPS) lumabri_proto.h luma
test_metrics: tests/c/test_metrics.c lumabri_metrics.h lumabri_stage_metrics.h
$(CC) $(CPPFLAGS) $(CFLAGS) tests/c/test_metrics.c -o $@

test_hybrid_parallel: tests/c/test_hybrid_parallel.c lumabri_client.h lumabri_proto.h lumabri_sign.h $(SECURE_DEPS)
test_hybrid_parallel: tests/c/test_hybrid_parallel.c src/runtime/lumabri_q8_reference.h lumabri_client.h lumabri_proto.h lumabri_sign.h $(SECURE_DEPS)
$(CC) $(CPPFLAGS) $(CFLAGS) -pthread tests/c/test_hybrid_parallel.c -o $@ -lm


Expand Down Expand Up @@ -592,6 +592,7 @@ test-sanitize:
$(CC) $(CPPFLAGS) $(SANITIZE_FLAGS) tests/c/test_content_filter.c -o build/sanitize/test_content_filter
$(CC) $(CPPFLAGS) $(SANITIZE_FLAGS) tests/c/test_scheduler.c -o build/sanitize/test_scheduler
$(CC) $(CPPFLAGS) $(SANITIZE_FLAGS) tests/c/test_planner_io.c -o build/sanitize/test_planner_io
$(CC) $(CPPFLAGS) $(SANITIZE_FLAGS) -pthread tests/c/test_hybrid_parallel.c -o build/sanitize/test_hybrid_parallel -lm
ASAN_OPTIONS=detect_leaks=1:halt_on_error=1 build/sanitize/test_segment_v2
ASAN_OPTIONS=detect_leaks=1:halt_on_error=1 build/sanitize/test_segment_discovery
ASAN_OPTIONS=detect_leaks=1:halt_on_error=1 build/sanitize/test_run_gate
Expand All @@ -600,6 +601,7 @@ test-sanitize:
ASAN_OPTIONS=detect_leaks=1:halt_on_error=1 build/sanitize/test_content_filter
ASAN_OPTIONS=detect_leaks=1:halt_on_error=1 build/sanitize/test_scheduler
ASAN_OPTIONS=detect_leaks=1:halt_on_error=1 build/sanitize/test_planner_io
ASAN_OPTIONS=detect_leaks=1:halt_on_error=1 build/sanitize/test_hybrid_parallel

test-thread-sanitize:
mkdir -p build/sanitize
Expand All @@ -623,7 +625,7 @@ SEGMENT_COMMON = segment_colibri.h src/runtime/lumabri_resident.h src/runtime/lu
lumabri_proto.h lumabri_sign.h lumabri_sha.h $(SECURE_DEPS)

HYBRID_PATCH_INPUTS = tools/prepare_resident_adapters.py engine_patches/make_patches.py \
lumabri_home_hybrid.h src/runtime/lumabri_hybrid_policy.h \
lumabri_home_hybrid.h src/runtime/lumabri_hybrid_policy.h src/runtime/lumabri_q8_reference.h \
engine_patches/deepseek_v4_p2p.py $(wildcard engine_patches/*-p2p.diff) \
lumabri_client.h lumibri_client.h lumi_v4_ext.h lumi_v4_bridge.c \
lumabri_proto.h lumabri_sign.h lumabri_secure.h lumabri_crypto.h \
Expand Down Expand Up @@ -665,7 +667,7 @@ $(COLIBRI_SEGMENT_LIB): $(HYBRID_ENGINE_DIR)/.prepared build/segment_hybrid_brid
$(AR) rcs $@ build/segment_hybrid_bridge.o

# Optional diagnostic; requires explicit, already approved household routes.
build/bench_home_hybrid: tests/c/bench_home_hybrid.c tests/c/bench_block_verify.h tests/c/bench_causal_spec.h lumabri_client.h lumabri_proto.h $(SECURE_DEPS) $(COLIBRI_SEGMENT_LIB)
build/bench_home_hybrid: tests/c/bench_home_hybrid.c tests/c/bench_hybrid_numeric.h tests/c/bench_block_verify.h tests/c/bench_causal_spec.h lumabri_client.h lumabri_proto.h $(SECURE_DEPS) $(COLIBRI_SEGMENT_LIB)
$(CC) $(CPPFLAGS) -O2 $(ENGINE_CPU_FLAGS) $(OMP_FLAGS) -pthread -I$(HYBRID_ENGINE_DIR) \
-include lumi_v4_ext.h -DLUMABRI_P2P -DLUMIBRI_P2P -DCOLI_SEGMENT_ADAPTER -DCOLI_EDGE_ADAPTER \
tests/c/bench_home_hybrid.c $(COLIBRI_SEGMENT_LIB) -o $@ -lm $(OMP_LIBS)
Expand Down
36 changes: 34 additions & 2 deletions docs/HYBRID_CONCURRENCY.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,13 +60,45 @@ the recipient key; the readiness probe also checks allocation, checkpoint and
numeric class. Public expert discovery, relay, hedges and implicit peer adoption
remain disabled. Older packages fail capability preflight before preparation.

For each covered layer, one selected expert is sent remotely while the other
top-k minus one execute locally, with unchanged router-order accumulation.
For each covered layer, the adaptive policy measures all-local and a remote
prefix of up to `min(top_k - 1, 8)` experts. The local suffix runs while remote
requests are in flight, with unchanged router-order accumulation. Exploration
is paced and reversible; see [Resident runtime](RESIDENT_RUNTIME.md) for its
cadence and timing semantics.
Uncovered layers stay local. Bounded RPC failures can recompute from the full
resident local copy. This is not a speedup guarantee: latency or a slow donor
can still dominate. `LUMABRI_HOME_HYBRID=0` selects normal Segment for A/B tests.
Other families keep their existing Segment implementation.

### Borderline numerical verification (OLMoE household only)

The normal pairwise FP32 check retains its `1e-6 + 1e-5*abs(reference)`
envelope. If a sampled finite pair fails that check but lies within twice
the larger per-value envelope, the OLMoE adapter can supply an independent
reference. The factor of two only bounds eligibility for extra work; it
never accepts an output. **Both** local and remote vectors must pass the
original envelope against the reference, or the existing local fallback
still applies. Remote output is retained unchanged after success.

The exceptional reference uses already-resident int8 weights and row scales,
long-double accumulation and stable SiLU, rounding its final result to FP32.
It cannot fetch weights. Missing slots, incompatible dimensions, other weight
formats, activation-quantized IDOT/FUSED3, a background pilot, fast-math and
failed/invalid references are ineligible. Other adapters without a reference
callback retain their original policy. This is a sampled interoperability
check, not proof of universal bitwise equivalence or adversarial correctness.

Reference attempts and accepts are counted separately; their entire cost is
included in validation and end-to-end wall time. Admission reserves one extra
FP32 output vector and at most 16 bytes per intermediate element for the
reference scratch. No new donor permission, weight copy, network request or
checkpoint read is introduced by this verification.

Tests cover the physical rounding counterexample, both sides failing their
independent bound, NaN/Inf, unavailable references, gross errors, cancellation
near zero, original no-reference behavior and resident-only adapter access.
The full-model greedy oracle remains a separate release gate.

Closing the chat retains approved RAM for another conversation. Owner Stop
unloads it. The private restart hint preserves the Hybrid label, but grants no
new authority: reconnection still checks host identity and checkpoint. Owner
Expand Down
Loading
Loading