Skip to content

fix(opencode-review): reach a healthy fallback model within the step budget#372

Closed
seonghobae wants to merge 4 commits into
mainfrom
fix/opencode-review-model-pool-fallback-budget
Closed

fix(opencode-review): reach a healthy fallback model within the step budget#372
seonghobae wants to merge 4 commits into
mainfrom
fix/opencode-review-model-pool-fallback-budget

Conversation

@seonghobae

Copy link
Copy Markdown
Contributor

Problem

The Required OpenCode Review check fails (red) on every open PR in ContextualWisdomLab/gyeot#4, #5, and #6, including #6 which is a one-file docs change. Because a tiny docs PR fails identically to a large one, this is a structural failure of the model-pool step, not a transient flake or a real code finding. All other required checks on those PRs (CodeQL, osv-scanner, trivy-fs, scorecard, dependency-review, verify) pass.

Root cause (from the actual job logs)

gh api repos/ContextualWisdomLab/gyeot/actions/jobs/<id>/logs on all three runs shows the identical sequence in Run OpenCode PR Review model pool:

Starting OpenCode model pool cycle 1.
OpenCode github-models/openai/o4-mini attempt 1/1 failed with exit 124.   # killed at 90-min timeout
OpenCode github-models/openai/o3-mini attempt 1/1 failed with exit 1.      # fast fail
OpenCode github-models/openai/gpt-5-mini attempt 1/1 failed with exit 124. # killed at 90-min timeout
OpenCode github-models/openai/gpt-5-nano attempt 1/1 failed with exit 124. # killed at 90-min timeout
##[error] The action 'Run OpenCode PR Review model pool' has timed out after 350 minutes.

exit 124 = the process was killed by the timeout wrapper at the 90-min (OPENCODE_RUN_TIMEOUT_SECONDS=5400) cap — i.e. the model hung, it did not fast-fail. The three front-loaded mini models each burn the full 90 min (~270 min total) before the pool can reach any high-quota fallback (deepseek/mistral/llama, candidates #6–11), so GitHub's hard 350-min step timeout kills the job first — opaquely, with no recorded review_status, which is why the check goes red instead of falling back.

At 90 min/model the pool mathematically cannot traverse enough of the 13-candidate list to reach a responsive model whenever ≥3 leading models hang — which is happening on every run.

Fix

  • OPENCODE_RUN_TIMEOUT_SECONDS 5400 → 1800 (30 min): still ample for a deep tool-using review (5× the 600 s the comment notes was "too short"), but low enough that the full 13-candidate list is traversable inside the 350-min step even in the worst case where every leading model hangs — so a responsive fallback is reached (worst case: candidate [codex] harden OpenCode agent evidence gate #11 llama-4-scout reached at 330 min < 340 min budget; in practice much sooner once a healthy model short-circuits).
  • OPENCODE_TOTAL_RETRY_BUDGET_SECONDS 0 → 20400 (340 min): a clean internal deadline 10 min under the step timeout. The pool already caps each run at min(run_timeout, deadline-now) and exits with an explicit retry deadline elapsed reason plus a recorded status, so a genuine exhaustion now prints its cause in the log instead of GitHub's opaque timed out after 350 minutes.
  • Updates the three scripts/ci/test_strix_quick_gate.sh assertions that pin these two values.

Verification

  • python3 -c yaml.safe_load(...) on the workflow — valid.
  • bash -n scripts/ci/test_strix_quick_gate.sh — valid.
  • pytest tests/test_opencode_workflow_shell_syntax.py — passes.
  • Budget math: 20400 / 1800 = 11 candidates reachable worst-case; llama-scout ([codex] harden OpenCode agent evidence gate #11) reached by minute 330, within the 340-min budget.

Notes / reconciliation

  • The workflow is org-central; gyeot runs it via the trusted OpenCode source ref. This PR fixes it at the correct locus (.github default branch); gyeot's inherited check turns green once this lands (a maintainer must re-pin if the trusted ref is a fixed SHA rather than main).
  • scripts/ci/test_strix_quick_gate.sh on main is already broadly out of sync with main's opencode-review.yml (~15 pre-existing assertion mismatches: model-ordering string, timeout-minutes: 285, event types:, etc.). It tracks an older pinned workflow version and is run against the trusted ref, not the PR head. This PR only reconciles the two timeout/budget assertions it directly changes; the remaining pre-existing drift is out of scope.

🤖 Generated with Claude Code

…budget

The Required OpenCode Review check timed out red on every gyeot PR
(ContextualWisdomLab/gyeot #4, #5, #6 — including a one-file docs PR),
so the model-pool step is structurally, not transiently, broken.

Root cause (from the job logs): the 3-4 front-loaded mini models
(o4-mini, gpt-5-mini, gpt-5-nano) each hang/rate-limit and are killed
at the 90-min (5400s) per-model `timeout` (exit 124). Three hangs burn
~270 min before the pool can reach any high-quota fallback
(deepseek/mistral/llama, candidates #6-11), so the 350-min hard step
timeout kills the job first — opaquely, with no recorded review_status.

Fix:
- OPENCODE_RUN_TIMEOUT_SECONDS 5400 -> 1800 (30 min): still ample for a
  deep tool-using review (5x the 600s that was too short), but low
  enough that the full 13-candidate list is traversable inside the
  350-min step even if every leading model hangs, so a responsive
  fallback is reached.
- OPENCODE_TOTAL_RETRY_BUDGET_SECONDS 0 -> 20400 (340 min): a clean
  internal deadline 10 min under the step timeout. The pool already
  caps each run at min(run_timeout, deadline-now) and exits with an
  explicit "retry deadline elapsed" reason + recorded status, so a
  genuine exhaustion now prints its cause instead of GitHub's opaque
  "timed out after 350 minutes".

Also updates the strix quick-gate assertions that pin these two values.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RTAMs4bpSZS77Xe3RQjv9P
@seonghobae

Copy link
Copy Markdown
Contributor Author

Superseded by #383. The latest review decision is that 30 minutes is still too short for OpenCode deep reviews; #383 keeps DeepSeek V3 first, restores a 90-minute per-model review window, and keeps a bounded pool budget so failures still publish a visible reason.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant