perf: make Rust core the primary numeric path (1e-6 parity gate)#115
Conversation
Software with both Python and Rust should run its numeric/mathematical computation on the Rust side. The Rust core (crates/mlsirm-core) already implements the verified LSIRM/MLS2PLM neg-loglik, gradients, and distance kernels (parity-verified in PR #114); this makes it the primary path. - FitConfig.backend and CLI --backend default numpy -> auto, so callers use the compiled fast_mlsirm._core (Rust/PyO3) kernel when available and fall back to the numpy reference otherwise. The numpy reference path is preserved for parity testing and fallback. - Add tests/test_rust_parity.py: Rust<->numpy parity gate asserting agreement to 1e-6 over all five model variants, four problem sizes, and masked/dense fixtures (observed max difference ~1e-13). Observable outputs are unchanged. - CI python job: add Rust toolchain and a resolved-default-backend assertion so the primary Rust path is built and exercised by pytest. - Update tests that assumed the numpy default; README and CHANGELOG. Formulas are ported bit-for-bit, not re-derived or optimized. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RTAMs4bpSZS77Xe3RQjv9P
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
# Conflicts: # .github/workflows/ci.yml
OpenCode Review Overview
Pull request overviewOpenCode reviewed the current-head bounded evidence and found no blocking issues. FindingsNo blocking findings. SummaryApproval sufficiency: bounded evidence supplied affirmative approval evidence for changed files, coverage/docstring posture, risk surfaces, and current-head verification; approval is not based merely on the absence of known blockers.
Changed-File Evidence Mapflowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Workflow: ci.yml"]
S1 --> I1["GitHub Actions review job"]
I1 --> R1["Review risk: Workflow: ci.yml"]
R1 --> V1["actionlint plus required checks"]
Evidence --> S2["Changed file (4 files)"]
S2 --> I2["repository behavior"]
I2 --> R2["Review risk: Changed file (4 files)"]
R2 --> V2["required checks"]
Evidence --> S3["Test (4 files)"]
S3 --> I3["regression suite"]
I3 --> R3["Review risk: Test (4 files)"]
R3 --> V3["targeted test run"]
|
![opencode-agent[bot]](https://avatars.githubusercontent.com/in/1549082?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
OpenCode reviewed the current-head bounded evidence and found no blocking issues.
Findings
No blocking findings.
Summary
Approval sufficiency: bounded evidence supplied affirmative approval evidence for changed files, coverage/docstring posture, risk surfaces, and current-head verification; approval is not based merely on the absence of known blockers.
Verification posture: CodeGraph evidence was initialized and bounded current-head evidence reviewed for changed-file evidence including .github/workflows/ci.yml, CHANGELOG.md, README.md, python/fast_mlsirm/cli.py, python/fast_mlsirm/config.py, and 4 more.
Linter/static: workflow/static review evidence is bounded by the current-head GitHub Checks gate and changed-file evidence.
TDD/regression: coverage execution evidence and focused changed hunks were reviewed from bounded-review-evidence.md.
Coverage: coverage execution evidence reports supported repository test suites passed.
Docstring coverage: coverage execution evidence reports configured repository docstring gates passed or docstring coverage was advisory.
DAG: CodeGraph/source-backed behavior map connects .github/workflows/ci.yml to the affected review, runtime, or workflow path and required checks.
PoC/execution: coverage-evidence job executed on the current head and reported PASS.
DDD/domain: workflow and repository-governance invariants were reviewed against changed files in bounded evidence.
CDD/context: CodeGraph evidence, changed-file history, and focused hunks were reviewed from bounded-review-evidence.md.
Similar issues: changed-file history evidence was reviewed for comparable local precedents.
Claim/concept check: bounded evidence, repository source, current-head workflow evidence, and, where numeric, scientific, statistical, or literature-backed claims are affected, original-paper/formula evidence and parameter-recovery expectations were used for claims.
Standards search: standards and external-source checks are delegated to configured OpenCode web_search/Context7/DeepWiki sources when applicable; no evidence-backed standards blocker is present in bounded evidence.
Compatibility/convention: changed workflow/script conventions, object naming, and reserved-word safety for schema/API/config/code surfaces were checked in bounded evidence.
Breaking-change/backcompat: deployment evidence and changed-file history were checked for backward-compatibility risk.
Performance: changed surfaces were checked for performance risk in bounded evidence.
Developer experience: changed automation, review, test, setup, and maintenance surfaces were checked for helpful or obstructive DX impact in bounded evidence.
User experience: connected user, operator, API, CLI, documentation, review-comment, status-check, rendering, and workflow-reader behavior was checked for contradictions against code, docs, and tests in bounded evidence.
Visual/DOM: Playwright visual, DOM locator, ARIA snapshot, console, and responsive evidence were checked when a web UI surface was present; for non-web surfaces, API/CLI/log/docs/workflow interaction evidence was reviewed instead.
Accessibility/i18n: accessibility, localization, and human-readable text surfaces were checked where UI, CLI, API message, docs, logs, or review text changed.
Supply-chain/license: dependency, package, model, container, and external-tool changes were checked in bounded evidence.
Packaging: package, build, test, lint, and security contracts were checked in bounded evidence.
Security/privacy: workflow-token, review-gate, and repository-automation security/privacy boundaries were checked in bounded evidence.
- Result: APPROVE
- Reason: The PR makes Rust the primary numeric path with 1e-6 parity gate, and all tests pass with high coverage.
- Head SHA:
dc6e1979ba85b8b4ed1e200a06bdb8fb30606428 - Workflow run: 29083804785
- Workflow attempt: 1
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Workflow: ci.yml"]
S1 --> I1["GitHub Actions review job"]
I1 --> R1["Review risk: Workflow: ci.yml"]
R1 --> V1["actionlint plus required checks"]
Evidence --> S2["Changed file (4 files)"]
S2 --> I2["repository behavior"]
I2 --> R2["Review risk: Changed file (4 files)"]
R2 --> V2["required checks"]
Evidence --> S3["Test (4 files)"]
S3 --> I3["regression suite"]
I3 --> R3["Review risk: Test (4 files)"]
R3 --> V3["targeted test run"]
|
Current-head verification for this loop:
|
2 participants
Summary
Software with both Python and Rust should run its numeric/mathematical computation on the Rust side. This PR makes the existing Rust core (
crates/mlsirm-core, exposed via thefast_mlsirm._corePyO3/maturin extension) the primary numeric path for the neg-loglik, gradients, and distance kernels. The pure-NumPy implementation inobjective.pyis retained as the reference/fallback.The verified LSIRM/MLS2PLM equations (parity-verified in #114) are ported bit-for-bit — not re-derived or "optimized". Observable outputs do not change.
What changed
FitConfig.backendand the CLI--backenddefault fromnumpy→auto, which resolves to the compiled Rust core when available and transparently falls back to the NumPy reference otherwise (backend.resolve_backend). The NumPy path is preserved for parity testing and as a fallback — nothing was deleted.tests/test_rust_parity.py. Asserts Rust ↔ NumPy agreement to a tight1e-6absolute tolerance over all five model variants (MLS2PLM,MLSRM,MIRT,ULS2PLM,ULSRM), four problem sizes, and both masked and dense fixtures — for objective, loglik, and every gradient block. Measured worst-case difference is ~1e-13, far inside the gate.pythonjob now installs the Rust toolchain and asserts the default backend resolves torust, so the primary path is built and exercised bypytest(not just the packaging job).numpydefault, plus README and CHANGELOG.Kernels moved to Rust (primary)
The genuine numeric computation —
neg_loglik_and_grad(softplus/sigmoid link, Euclidean distance kernel, and analytic gradients fortheta,alpha,b,xi,zeta,tau, plus L2 penalties) — now runs in Rust by default. Pure orchestration / IO-bound code (optimizer loops, packing, simulation IO) is left in Python by design: it is not numeric-heavy and moving it would not help.Verification
pytest: 172 passed (was 131; +41 parity fixtures).cargo test --workspaceand the PyO3 crate tests: green.scripts/release_acceptance.py --require-rustsmoke:status: ok.1e-6gate.Constraints honored
1e-6; observable results unchanged.🤖 Generated with Claude Code