Skip to content

Raise autotune process timeout from 30s to 7260s (2h1m)#396

Merged
Augustas11 merged 11 commits into
mainfrom
fix/autotune-timeout-progress
Jul 5, 2026
Merged

Raise autotune process timeout from 30s to 7260s (2h1m)#396
Augustas11 merged 11 commits into
mainfrom
fix/autotune-timeout-progress

Conversation

@Augustas11

Copy link
Copy Markdown
Owner

Summary

Raise AutotuneRecommendationRunner.processTimeout from 30s to 7260s (2h1m) so fresh-install onboarding actually reaches .live on real hardware.

Root cause: The 30s value in AutotuneRecommendationRunner.swift was ~240× too short. Every fresh-install v1.8.3 smoke on this Mac hit AutotuneRecommendationError.timedOut at T+30s while autotune was still spawning MLX probes. Autotune runs Stage-1 benchmarks against every non-blocked, RAM-eligible catalog row — each candidate takes up to 720s (readyTimeout 120s + prewarm probe 300s + measured probe 300s per SPEC-023 v1.7.5), catalog cardinality scales with Mac tier, and the CLI --recommend path does NOT enforce its own declared maxDuration=7200s. So the App-side timeout IS the authoritative ceiling for autotune runtime.

Value: 7260s = realistic worst-case autotune (10 candidates × 720s = 7200s) + 60s grace. Below 2.5h untenable-spinner ceiling. Median UX unaffected (empirical: 2-5 min).

Files

  • phase3-binary/app/Sources/Malibu/Onboarding/AutotuneRecommendationRunner.swift — bump processTimeout 30→7260s, visibility private staticstatic so tests can pin the invariant, rewrite rationale comment with 4 numbered reasons for the value and the honest note that the CLI --recommend path doesn't enforce its own maxDuration.
  • phase3-binary/app/Tests/MalibuTests/AutotuneRecommendationRunnerTimeoutTests.swift — new test file, 2 tests pinning: (1) processTimeout ≥ realisticWorstCaseAutotuneSec (7200s); (2) processTimeout ≤ untenableSpinnerCeilingSec (2.5h).
  • specs/AUDIT_FIX_AUTOTUNE_TIMEOUT_{COMMON,CODE,SECURITY,ARCHITECT}_PROMPT.md — 3-lane codex audit prompts kept in-tree for future reference and R2-R5 convergence history.

Audit convergence (5 rounds, R5 final)

Round Lane Verdict Findings
R1 SEC READY L=1 (unbounded stdout buffer, still LOW at final 7260s per R5)
R1 ARCH READY L=1 (obsoleted by R2+ test rewrite)
R1 CODE NEEDS REVISION M-1: per-candidate math missed prewarm probeOnce step
R2 CODE NEEDS REVISION M-1: candidate count is not fixed at 2-3, --recommend iterates all non-blocked RAM-eligible rows; M-2: pre-existing CLI-side orphan-child bug (deferred, see below)
R3 CODE NEEDS REVISION M-1: rationale claimed 7260s "mirrors CLI cap", but --recommend doesn't enforce maxDuration
R4 CODE READY L=1 (wording typo, fixed)
R5 CODE READY 0/0/0/0
R5 SEC READY L=1 (SEC-L-1 unbounded stdout buffer, see below)
R5 ARCH READY L=3 (ARCH-L-1/2/3, see below)

All CRITICAL / HIGH / MEDIUM findings converged to zero.

LOW findings — documented, deferred

Per project convention: LOWs ship with PR-body docs.

  • SEC-L-1ProcessOutputBuffer at AutotuneRecommendationRunner.swift:33-50 is unbounded. Realistic autotune stdout is single-digit MB (per-probe JSON + final result). Would need a compromised/spoofed CLI to matter. Local DoS/OOM only.
  • ARCH-L-1 — CLI --recommend path at phase3-binary/Sources/macprovider-cli/AutotuneCommand.swift:131-139 does NOT enforce declared maxDuration=7200s (deadline is only installed in non-recommend at :157-161). CLI-side fix belongs in a follow-up PR.
  • ARCH-L-2 — 10-candidate floor in the App test is a defensible current-policy assumption, not a real enforced catalog upper bound. If the catalog ever admits >10 candidates on some Mac tier, this timeout gets tight; heartbeat protocol becomes necessary before that point.
  • ARCH-L-3AutotuneRecommendationError.timedOut copy is generic. At 2h1m, human failure copy per SPEC-026 §6.4 should convey the wait was long. Follow-up.

Deferred to CLI-side follow-up

  • R2 CODE-M-2 — Orphan child macprovider-cli serve subprocess when App-side timeout fires. Stage1Prober spawns the serve child at Stage1Iterator.swift:436-442 and relies on defer { runner.stop(...) } (:443-445). The CLI --recommend path does NOT install signal handlers (only non-recommend does at AutotuneCommand.swift:157-161), so process.terminate() from the App kills the parent CLI without giving Stage1Prober a cleanup path. The child leaks and can hold port 18080 + model memory. This is a pre-existing CLI-side bug that exists at any App-side timeout value ≥ 0; not introduced by this PR. Fix belongs in phase3-binary/Sources/macprovider-cli/AutotuneCommand.swift to install signal handlers on the recommend path.

Test plan

  • MalibuTests: 91/91 pass locally (xcodebuild -project Malibu.xcodeproj -scheme MalibuTests test)
  • 3-lane codex audit converged to 0 CRITICAL / 0 HIGH / 0 MEDIUM across all lanes
  • Post-merge: bump binaryVersion 1.8.3 → 1.8.4, dispatch v1.8.4 prerelease
  • Post-merge: clean-slate install v1.8.4 on this Mac, drive Malibu.app onboarding through the state machine, verify .live end-to-end

🤖 Generated with Claude Code

Augustas11 and others added 10 commits July 5, 2026 14:11
Fresh-install onboarding was 100% breaking at .autotuning because
AutotuneRecommendationRunner.processTimeout=30s was ~60× too short
vs. the CLI's actual Stage-1 probe budget.

Autotune runs benchmarks against 2-3 candidate models. Each candidate
spawns a macprovider-cli serve subprocess, waits for MLX to load
model weights (30-60s), runs a prewarm probe (60-180s prefill on
M-Base 30B MoE per SPEC-023 v1.7.5), then measures TTFT + tokens/sec.
Total per-candidate wall-clock is 60-420s.

30s budget killed the CLI mid-model-load on every fresh install →
.failed(autotuning, retryable: true, AutotuneRecommendationError.timedOut)
loop, retry hit same failure.

Bump to 1800s (30 min). Cover 3 × 420s CLI worst case + margin.
60-min upper-bound guard so wedged subprocesses don't hang the UI
forever.

Added AutotuneRecommendationRunnerTimeoutTests pinning the invariant
against the CLI's Stage1Prober timings (readyTimeoutSec=120,
probeIdleTimeoutSec=300 in Stage1Iterator.swift) so future timeout
changes have to justify themselves against the CLI's own budget.

Repro / rationale in the 2026-07-05 smoke report at:
/private/tmp/claude-501/.../scratchpad/smoke-v183/

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Prompts for CODE / SECURITY / ARCHITECT lanes against ae23d48.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
R1 audit (CODE lane) caught undercount in per-candidate budget.

Stage1Prober runs THREE HTTP calls per candidate, not two:
- readyTimeoutSec = 120s (MLX model load)
- prewarm probeOnce = probeIdleTimeoutSec = 300s (Track A3, :474)
- measured replicate probeOnce = probeIdleTimeoutSec = 300s (default
  stage1Replicates=1, AutotuneCommand.swift:35-37)

So per-candidate worst case = 120 + 300 + 300 = 720s (not 420s).
3-candidate worst case = 2160s (not 1260s).

Bump processTimeout: 1800s -> 2700s (45 min) to cover 2160s + ~25%
margin, still well below CLI outer maxDuration=7200s so a healthy CLI
fails fast internally before this deadline fires.

Update rationale comment + test constant cliPerCandidateWorstCaseSec
420 -> 720. 60-min upper-bound test guard unchanged.

Tests: 92 pass, 0 fail (MalibuTests.xctest).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
R2 CODE audit found the per-candidate math frame was wrong. Autotune
--recommend iterates ALL non-blocked RAM-eligible catalog rows, not
a fixed 2-3 set. On a 36GB Mac that's 4 rows × 720s = 2880s > 2700s
cap; larger Macs hit 5+.

Any per-candidate × N estimate under-counts on some hardware tier.
The only principled ceiling is the CLI's OWN outer hard budget at
AutotuneCommand.swift:47-48 (maxDuration = 7200s / 2h).

Set processTimeout = 7200 + 60s grace = 7260s so:
- Normal case (2-5 min empirically): CLI returns first, App-side
  timer never fires.
- CLI hits its own maxDuration: fails-fast with nonzero exit before
  App-side timer fires.
- CLI truly wedged past 2h: App fires as last-line-of-defense
  cleanup.

Rewrite tests to pin the invariant against CLI's declared
maxDuration + grace, not against a guess at candidate cardinality.
Upper-bound guard raised to 2.5h.

CODE-M-2 (orphan child serve subprocess on SIGTERM) is a
pre-existing CLI-side bug unaffected by this timeout value — filed
as follow-up in PR body. The --recommend path doesn't install
signal handlers so App-side process.terminate() can't cascade
cleanup; fix belongs in AutotuneCommand.swift.

Tests: 91 pass, 0 fail (dropped one obsolete candidate-math test,
added two principled invariants).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
R3 CODE audit found the R2 rationale lied: the CLI's --recommend
path does NOT enforce its declared maxDuration=7200s. The deadline
is only created in the non-recommend path at
AutotuneCommand.swift:157-161. runAutotuneRecommend() at :131-139
returns before that and calls
AutotuneRecommendationBenchmarker().benchmarks() with no deadline.

So 7260s is not "CLI cap + grace" — it IS the outer bound of
autotune runtime, chosen App-side.

Value unchanged (7260s is still correct). Rewrite:
- Rationale comment: honest description of App-side authoritative
  ceiling with 4 concrete reasons the value is 7260s.
- Test invariants: pin to "realistic worst-case autotune runtime"
  (7200s = 10 candidates × 720s ceiling) NOT to a mirrored CLI cap
  that doesn't exist. Upper bound still 2.5h untenable-spinner
  guard.

Tests: pass locally.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

@antfleet-ops antfleet-ops left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Autotune timeout fix. 5 rounds of 3-lane codex audit; all lanes converged 0 C/H/M. LOWs documented. R2 CODE-M-2 orphan child correctly deferred to CLI-side follow-up. Approved.

@Augustas11 Augustas11 merged commit 457313d into main Jul 5, 2026
13 checks passed
@Augustas11 Augustas11 deleted the fix/autotune-timeout-progress branch July 5, 2026 12:01
Augustas11 added a commit that referenced this pull request Jul 5, 2026
Cadence bump for smoke re-run of PR #396 (autotune process timeout
30s -> 7260s / 2h1m). v1.8.4 prerelease is the first .pkg containing
the timeout fix.

Grepped for '1.8.3' in both source + tests per
[feedback-version-bump-grep-both-versions]. Historical 1.8.3 docstring
references in AutotuneRecommendationRunner.swift and
AutotuneRecommendationRunnerTimeoutTests.swift are intentional
records of the smoke failure that motivated the fix and stay pinned
to 1.8.3.

coordinator.yaml latest_binary_version stays at 1.8.3 until v1.8.4
prerelease validation confirms the fix — follow-up PR bumps that.

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants