Skip to content

feat: add IRT stability evidence helpers#124

Merged
seonghobae merged 4 commits into
mainfrom
codex/science-stability
Jul 10, 2026
Merged

feat: add IRT stability evidence helpers#124
seonghobae merged 4 commits into
mainfrom
codex/science-stability

Conversation

@seonghobae

@seonghobae seonghobae commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Allow missing-by-design all-missing person/item axes while preserving the global "at least one observed response" guard.
  • Add finite diagnostics for empty axes plus regression coverage for missing values, 0/full-score initialization, true-parameter reproduction, Hessian/vcov/standard errors, second-order checks, fixed item parameter linking, CAT item selection, and greedy ATA constraints.
  • Add aFIPC-style fixed-item calibration diagnostics and CLI support for kaefa-style item-fit penalized candidate selection. Addresses fast-mlsirm: aFIPC Fixed-Item Parameter Calibration + kaefa item-fit #113 after merge.
  • Add Product Design/Figma handoff for IRT Stability Review and fixed-item calibration review screens: Information Architecture, screen definition, key screen, wireframe, user stories, and evidence-sync frame coverage.

Scientific basis

  • Keeps the existing MLS2PLM simple-structure equation contract intact.
  • Uses observed-information Hessian, vcov, standard-error, and positive-definite second-order checks in the same spirit as mirt SE/vcov/secondOrderTest workflows.
  • Adds fixed-item linking, CAT, ATA, and fixed-item calibration support as minimal local helpers without adding new dependencies.

Merge order

This PR must wait behind lower-numbered open PRs, especially #18. Do not merge this ahead of the current ascending queue or without current-head OpenCode approval and required checks.

Verification

  • py -m pytest tests/test_irt_stability.py tests/test_objective.py tests/test_diagnostics.py tests/test_cli.py tests/test_figma_evidence_sync.py -q -> 51 passed, 2 skipped
  • py -m pytest tests -q -> 138 passed, 3 skipped
  • cargo test --workspace -> not run locally because cargo is not installed on this Windows PATH

@opencode-agent

opencode-agent Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

OpenCode Review Overview

  • Head SHA: 2c90a7ae08b848315119cc864b888353ff230006
  • Workflow run: 29085231902
  • Workflow attempt: 1
  • Gate result: APPROVE (approval step)

Pull request overview

OpenCode reviewed the current-head bounded evidence and found no blocking issues.

Findings

No blocking findings.

Summary

Approval sufficiency: bounded evidence supplied affirmative approval evidence for changed files, coverage/docstring posture, risk surfaces, and current-head verification; approval is not based merely on the absence of known blockers.
Verification posture: CodeGraph evidence was initialized and bounded current-head evidence reviewed for changed-file evidence including CHANGELOG.md, README.md, docs/buyer_demo_storyboard.md, docs/figma_product_design_packet.md, docs/irt_stability_product_design.md, and 14 more.
Linter/static: workflow/static review evidence is bounded by the current-head GitHub Checks gate and changed-file evidence.
TDD/regression: coverage execution evidence and focused changed hunks were reviewed from bounded-review-evidence.md.
Coverage: coverage execution evidence reports supported repository test suites passed.
Docstring coverage: coverage execution evidence reports configured repository docstring gates passed or docstring coverage was advisory.
DAG: CodeGraph/source-backed behavior map connects CHANGELOG.md to the affected review, runtime, or workflow path and required checks.
PoC/execution: coverage-evidence job executed on the current head and reported PASS.
DDD/domain: workflow and repository-governance invariants were reviewed against changed files in bounded evidence.
CDD/context: CodeGraph evidence, changed-file history, and focused hunks were reviewed from bounded-review-evidence.md.
Similar issues: changed-file history evidence was reviewed for comparable local precedents.
Claim/concept check: bounded evidence, repository source, current-head workflow evidence, and, where numeric, scientific, statistical, or literature-backed claims are affected, original-paper/formula evidence and parameter-recovery expectations were used for claims.
Standards search: standards and external-source checks are delegated to configured OpenCode web_search/Context7/DeepWiki sources when applicable; no evidence-backed standards blocker is present in bounded evidence.
Compatibility/convention: changed workflow/script conventions, object naming, and reserved-word safety for schema/API/config/code surfaces were checked in bounded evidence.
Breaking-change/backcompat: deployment evidence and changed-file history were checked for backward-compatibility risk.
Performance: changed surfaces were checked for performance risk in bounded evidence.
Developer experience: changed automation, review, test, setup, and maintenance surfaces were checked for helpful or obstructive DX impact in bounded evidence.
User experience: connected user, operator, API, CLI, documentation, review-comment, status-check, rendering, and workflow-reader behavior was checked for contradictions against code, docs, and tests in bounded evidence.
Visual/DOM: Playwright visual, DOM locator, ARIA snapshot, console, and responsive evidence were checked when a web UI surface was present; for non-web surfaces, API/CLI/log/docs/workflow interaction evidence was reviewed instead.
Accessibility/i18n: accessibility, localization, and human-readable text surfaces were checked where UI, CLI, API message, docs, logs, or review text changed.
Supply-chain/license: dependency, package, model, container, and external-tool changes were checked in bounded evidence.
Packaging: package, build, test, lint, and security contracts were checked in bounded evidence.
Security/privacy: workflow-token, review-gate, and repository-automation security/privacy boundaries were checked in bounded evidence.

  • Result: APPROVE
  • Reason: No blocking issues found in the PR changes.
  • Head SHA: 2c90a7ae08b848315119cc864b888353ff230006
  • Workflow run: 29085231902
  • Workflow attempt: 1

Changed-File Evidence Map

flowchart LR
  PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
  Evidence --> S1["Changed file (10 files)"]
  S1 --> I1["repository behavior"]
  I1 --> R1["Review risk: Changed file (10 files)"]
  R1 --> V1["required checks"]
  Evidence --> S2["Docs (3 files)"]
  S2 --> I2["operator or user guidance"]
  I2 --> R2["Review risk: Docs (3 files)"]
  R2 --> V2["docs review"]
  Evidence --> S3["Test (6 files)"]
  S3 --> I3["regression suite"]
  I3 --> R3["Review risk: Test (6 files)"]
  R3 --> V3["targeted test run"]
Loading

@opencode-agent opencode-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

OpenCode reviewed the current-head bounded evidence and found no blocking issues.

Findings

No blocking findings.

Summary

Approval sufficiency: bounded evidence supplied affirmative approval evidence for changed files, coverage/docstring posture, risk surfaces, and current-head verification; approval is not based merely on the absence of known blockers.
Verification posture: CodeGraph evidence was initialized and bounded current-head evidence reviewed for changed-file evidence including CHANGELOG.md, README.md, docs/buyer_demo_storyboard.md, docs/figma_product_design_packet.md, docs/irt_stability_product_design.md, and 14 more.
Linter/static: workflow/static review evidence is bounded by the current-head GitHub Checks gate and changed-file evidence.
TDD/regression: coverage execution evidence and focused changed hunks were reviewed from bounded-review-evidence.md.
Coverage: coverage execution evidence reports supported repository test suites passed.
Docstring coverage: coverage execution evidence reports configured repository docstring gates passed or docstring coverage was advisory.
DAG: CodeGraph/source-backed behavior map connects CHANGELOG.md to the affected review, runtime, or workflow path and required checks.
PoC/execution: coverage-evidence job executed on the current head and reported PASS.
DDD/domain: workflow and repository-governance invariants were reviewed against changed files in bounded evidence.
CDD/context: CodeGraph evidence, changed-file history, and focused hunks were reviewed from bounded-review-evidence.md.
Similar issues: changed-file history evidence was reviewed for comparable local precedents.
Claim/concept check: bounded evidence, repository source, current-head workflow evidence, and, where numeric, scientific, statistical, or literature-backed claims are affected, original-paper/formula evidence and parameter-recovery expectations were used for claims.
Standards search: standards and external-source checks are delegated to configured OpenCode web_search/Context7/DeepWiki sources when applicable; no evidence-backed standards blocker is present in bounded evidence.
Compatibility/convention: changed workflow/script conventions, object naming, and reserved-word safety for schema/API/config/code surfaces were checked in bounded evidence.
Breaking-change/backcompat: deployment evidence and changed-file history were checked for backward-compatibility risk.
Performance: changed surfaces were checked for performance risk in bounded evidence.
Developer experience: changed automation, review, test, setup, and maintenance surfaces were checked for helpful or obstructive DX impact in bounded evidence.
User experience: connected user, operator, API, CLI, documentation, review-comment, status-check, rendering, and workflow-reader behavior was checked for contradictions against code, docs, and tests in bounded evidence.
Visual/DOM: Playwright visual, DOM locator, ARIA snapshot, console, and responsive evidence were checked when a web UI surface was present; for non-web surfaces, API/CLI/log/docs/workflow interaction evidence was reviewed instead.
Accessibility/i18n: accessibility, localization, and human-readable text surfaces were checked where UI, CLI, API message, docs, logs, or review text changed.
Supply-chain/license: dependency, package, model, container, and external-tool changes were checked in bounded evidence.
Packaging: package, build, test, lint, and security contracts were checked in bounded evidence.
Security/privacy: workflow-token, review-gate, and repository-automation security/privacy boundaries were checked in bounded evidence.

  • Result: APPROVE
  • Reason: All tests pass with high coverage, and no unresolved issues or security concerns were identified.
  • Head SHA: a676dbafb2cce957339fab4fb6f82b4d686629e8
  • Workflow run: 29084556121
  • Workflow attempt: 1

Changed-File Evidence Map

flowchart LR
  PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
  Evidence --> S1["Changed file (10 files)"]
  S1 --> I1["repository behavior"]
  I1 --> R1["Review risk: Changed file (10 files)"]
  R1 --> V1["required checks"]
  Evidence --> S2["Docs (3 files)"]
  S2 --> I2["operator or user guidance"]
  I2 --> R2["Review risk: Docs (3 files)"]
  R2 --> V2["docs review"]
  Evidence --> S3["Test (6 files)"]
  S3 --> I3["regression suite"]
  I3 --> R3["Review risk: Test (6 files)"]
  R3 --> V3["targeted test run"]
Loading

@seonghobae

Copy link
Copy Markdown
Contributor Author

Current-head verification for this loop:

  • Head: a676dba
  • Action taken: merged current main into codex/science-stability and pushed a refreshed current HEAD.
  • Codegraph: codegraph init confirmed the repository was initialized; codegraph index re-indexed 54 files.
  • Local verification: py -m pytest tests/test_irt_stability.py tests/test_diagnostics.py tests/test_objective.py tests/test_cli.py -q => 48 passed, 2 skipped; py -m pytest -q => 141 passed, 3 skipped; git diff --check clean.
  • IRT/science surface verified: missing-by-design robustness, true-parameter reproduction, Hessian/vcov/standard-error/second-order tests, fixed-item linking, CAT/ATA helper coverage.
  • GitHub Checks on current HEAD: CI python/rust/package, CodeQL, Security Scan dependency-review/trivy-fs/osv-scan/scorecard, Strix, coverage-evidence, and OpenCode review all passed.
  • Security alerts checked: open code-scanning alerts = 0; open Dependabot alerts = 0.
  • Review surface checked: inline review comments = 0; review threads = 0; OpenCode approved current HEAD.
  • Old-head queue checked: active old-head runs = 0.
  • Auto-merge requested after this verification. If merge is still held, the remaining blocker is repository/org native branch governance, not a build/security/test failure.

@opencode-agent opencode-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

OpenCode reviewed the current-head bounded evidence and found no blocking issues.

Findings

No blocking findings.

Summary

Approval sufficiency: bounded evidence supplied affirmative approval evidence for changed files, coverage/docstring posture, risk surfaces, and current-head verification; approval is not based merely on the absence of known blockers.
Verification posture: CodeGraph evidence was initialized and bounded current-head evidence reviewed for changed-file evidence including CHANGELOG.md, README.md, docs/buyer_demo_storyboard.md, docs/figma_product_design_packet.md, docs/irt_stability_product_design.md, and 14 more.
Linter/static: workflow/static review evidence is bounded by the current-head GitHub Checks gate and changed-file evidence.
TDD/regression: coverage execution evidence and focused changed hunks were reviewed from bounded-review-evidence.md.
Coverage: coverage execution evidence reports supported repository test suites passed.
Docstring coverage: coverage execution evidence reports configured repository docstring gates passed or docstring coverage was advisory.
DAG: CodeGraph/source-backed behavior map connects CHANGELOG.md to the affected review, runtime, or workflow path and required checks.
PoC/execution: coverage-evidence job executed on the current head and reported PASS.
DDD/domain: workflow and repository-governance invariants were reviewed against changed files in bounded evidence.
CDD/context: CodeGraph evidence, changed-file history, and focused hunks were reviewed from bounded-review-evidence.md.
Similar issues: changed-file history evidence was reviewed for comparable local precedents.
Claim/concept check: bounded evidence, repository source, current-head workflow evidence, and, where numeric, scientific, statistical, or literature-backed claims are affected, original-paper/formula evidence and parameter-recovery expectations were used for claims.
Standards search: standards and external-source checks are delegated to configured OpenCode web_search/Context7/DeepWiki sources when applicable; no evidence-backed standards blocker is present in bounded evidence.
Compatibility/convention: changed workflow/script conventions, object naming, and reserved-word safety for schema/API/config/code surfaces were checked in bounded evidence.
Breaking-change/backcompat: deployment evidence and changed-file history were checked for backward-compatibility risk.
Performance: changed surfaces were checked for performance risk in bounded evidence.
Developer experience: changed automation, review, test, setup, and maintenance surfaces were checked for helpful or obstructive DX impact in bounded evidence.
User experience: connected user, operator, API, CLI, documentation, review-comment, status-check, rendering, and workflow-reader behavior was checked for contradictions against code, docs, and tests in bounded evidence.
Visual/DOM: Playwright visual, DOM locator, ARIA snapshot, console, and responsive evidence were checked when a web UI surface was present; for non-web surfaces, API/CLI/log/docs/workflow interaction evidence was reviewed instead.
Accessibility/i18n: accessibility, localization, and human-readable text surfaces were checked where UI, CLI, API message, docs, logs, or review text changed.
Supply-chain/license: dependency, package, model, container, and external-tool changes were checked in bounded evidence.
Packaging: package, build, test, lint, and security contracts were checked in bounded evidence.
Security/privacy: workflow-token, review-gate, and repository-automation security/privacy boundaries were checked in bounded evidence.

  • Result: APPROVE
  • Reason: No blocking issues found in the PR changes.
  • Head SHA: 2c90a7ae08b848315119cc864b888353ff230006
  • Workflow run: 29085231902
  • Workflow attempt: 1

Changed-File Evidence Map

flowchart LR
  PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
  Evidence --> S1["Changed file (10 files)"]
  S1 --> I1["repository behavior"]
  I1 --> R1["Review risk: Changed file (10 files)"]
  R1 --> V1["required checks"]
  Evidence --> S2["Docs (3 files)"]
  S2 --> I2["operator or user guidance"]
  I2 --> R2["Review risk: Docs (3 files)"]
  R2 --> V2["docs review"]
  Evidence --> S3["Test (6 files)"]
  S3 --> I3["regression suite"]
  I3 --> R3["Review risk: Test (6 files)"]
  R3 --> V3["targeted test run"]
Loading

@seonghobae

Copy link
Copy Markdown
Contributor Author

Current-head verification after base refresh:

  • Head: 2c90a7a
  • Local verification: py -m pytest tests/test_irt_stability.py tests/test_diagnostics.py tests/test_objective.py tests/test_cli.py -q -> 48 passed, 2 skipped; py -m pytest -q -> 141 passed, 3 skipped; git diff --check clean.
  • Codegraph: initialized and re-indexed this loop.
  • GitHub Checks: CI python/rust/package, CodeQL, Security Scan trivy-fs/osv-scan/dependency-review/scorecard, OpenCode coverage-evidence/opencode-review, and Strix all passed on current HEAD.
  • Security alerts: open code scanning alerts = 0; open Dependabot alerts = 0.
  • Review sweep: review threads = 0; inline review comments = 0.

This PR is approved and clean, so I am proceeding with merge/auto-merge queue handling.

@seonghobae seonghobae merged commit 4617f22 into main Jul 10, 2026
25 checks passed
@seonghobae seonghobae deleted the codex/science-stability branch July 10, 2026 10:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant