Skip to content

perf(hybrid): tune approved remote expert fanout per layer - #181

Draft
JustVugg wants to merge 1 commit into
feat/household-parallel-performancefrom
feat/hybrid-measured-fanout
Draft

JustVugg wants to merge 1 commit into
feat/household-parallel-performancefrom
feat/hybrid-measured-fanout

Conversation

@JustVugg

Copy link
Copy Markdown
Owner

Summary

Stacked on #177 (feat/household-parallel-performance). This PR changes only the Hybrid workload split and its validation, not donor permissions, model arithmetic, resident allocation sizes, Colibri sources, or the wire protocol.

  • Replace the fixed one-remote-expert decision with measured per-layer counts from zero through min(top_k - 1, 8). At least one expert stays local. Existing approved resident routes remain the only eligible donors.
  • Compare two samples per count with interleaved local baselines, retain a 3% improvement threshold, periodically revisit alternatives, and cool down failed counts. Exploration has an explicit bounded round count and a real latency cost; this is not a monotonic speed guarantee.
  • Dispatch remote contributions before local computation and preserve router-order accumulation and all-or-nothing publication. Keep numerical probes, pinned identity checks and resident local fallback.
  • Add forced-count diagnostics, independent-state fanout sweeps, and full-model oracle accounting that rejects missing contributions or silent fallback.
  • Add a packaged household CI test with a private tiny OLMoE fixture, top-k=4, and verified three-remote/one-local work. Include it in Linux and native macOS jobs.

Locally passed

  • test_hybrid_parallel with -Wall -Wextra -Werror: policy choices for top-k 1..64, socket dispatch for remote counts 1..8, canonical accumulation, partial-failure cleanup, configuration limits, numerical rejection and authenticated reconnect.
  • The same test under AddressSanitizer and UndefinedBehaviorSanitizer with leak detection.
  • Accumulation-order, local-fallback and verified-failover regression tests.
  • test_home_expert: allocation/root/layer/shape/numeric-class/admission checks.
  • Full resident household flow with forced 3-remote/1-local, then adaptive mode on a four-expert tiny fixture: donor consent, approved execution, streamed chat, second private conversation, retained weights with source offline, no mirror writes, cleanup.
  • Household runtime and benchmark build against unmodified Colibri 12a5c464b5c1f8292d578c62458706bc32d6ac95.

Performance is deliberately not certified

A physical PC + Intel Mac service sweep used the existing approved resident allocation and exercised multiple remote counts. It was stopped when another inference process was confirmed actively consuming several PC cores. Those timings are contaminated and are not a speedup claim. The operator cannot stop that workload now.

Pending: clean physical native-local vs fanout vs adaptive sweep, followed by the full-model greedy-token oracle and end-to-end decode comparison. No claim of 8 tok/s, guaranteed speedup per donor, or production performance certification. CI must pass before merge; do not squash the dependent branches.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant