Raise autotune process timeout from 30s to 7260s (2h1m)#396
Merged
Conversation
Fresh-install onboarding was 100% breaking at .autotuning because AutotuneRecommendationRunner.processTimeout=30s was ~60× too short vs. the CLI's actual Stage-1 probe budget. Autotune runs benchmarks against 2-3 candidate models. Each candidate spawns a macprovider-cli serve subprocess, waits for MLX to load model weights (30-60s), runs a prewarm probe (60-180s prefill on M-Base 30B MoE per SPEC-023 v1.7.5), then measures TTFT + tokens/sec. Total per-candidate wall-clock is 60-420s. 30s budget killed the CLI mid-model-load on every fresh install → .failed(autotuning, retryable: true, AutotuneRecommendationError.timedOut) loop, retry hit same failure. Bump to 1800s (30 min). Cover 3 × 420s CLI worst case + margin. 60-min upper-bound guard so wedged subprocesses don't hang the UI forever. Added AutotuneRecommendationRunnerTimeoutTests pinning the invariant against the CLI's Stage1Prober timings (readyTimeoutSec=120, probeIdleTimeoutSec=300 in Stage1Iterator.swift) so future timeout changes have to justify themselves against the CLI's own budget. Repro / rationale in the 2026-07-05 smoke report at: /private/tmp/claude-501/.../scratchpad/smoke-v183/ Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Prompts for CODE / SECURITY / ARCHITECT lanes against ae23d48. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
R1 audit (CODE lane) caught undercount in per-candidate budget. Stage1Prober runs THREE HTTP calls per candidate, not two: - readyTimeoutSec = 120s (MLX model load) - prewarm probeOnce = probeIdleTimeoutSec = 300s (Track A3, :474) - measured replicate probeOnce = probeIdleTimeoutSec = 300s (default stage1Replicates=1, AutotuneCommand.swift:35-37) So per-candidate worst case = 120 + 300 + 300 = 720s (not 420s). 3-candidate worst case = 2160s (not 1260s). Bump processTimeout: 1800s -> 2700s (45 min) to cover 2160s + ~25% margin, still well below CLI outer maxDuration=7200s so a healthy CLI fails fast internally before this deadline fires. Update rationale comment + test constant cliPerCandidateWorstCaseSec 420 -> 720. 60-min upper-bound test guard unchanged. Tests: 92 pass, 0 fail (MalibuTests.xctest). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
R2 CODE audit found the per-candidate math frame was wrong. Autotune --recommend iterates ALL non-blocked RAM-eligible catalog rows, not a fixed 2-3 set. On a 36GB Mac that's 4 rows × 720s = 2880s > 2700s cap; larger Macs hit 5+. Any per-candidate × N estimate under-counts on some hardware tier. The only principled ceiling is the CLI's OWN outer hard budget at AutotuneCommand.swift:47-48 (maxDuration = 7200s / 2h). Set processTimeout = 7200 + 60s grace = 7260s so: - Normal case (2-5 min empirically): CLI returns first, App-side timer never fires. - CLI hits its own maxDuration: fails-fast with nonzero exit before App-side timer fires. - CLI truly wedged past 2h: App fires as last-line-of-defense cleanup. Rewrite tests to pin the invariant against CLI's declared maxDuration + grace, not against a guess at candidate cardinality. Upper-bound guard raised to 2.5h. CODE-M-2 (orphan child serve subprocess on SIGTERM) is a pre-existing CLI-side bug unaffected by this timeout value — filed as follow-up in PR body. The --recommend path doesn't install signal handlers so App-side process.terminate() can't cascade cleanup; fix belongs in AutotuneCommand.swift. Tests: 91 pass, 0 fail (dropped one obsolete candidate-math test, added two principled invariants). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
R3 CODE audit found the R2 rationale lied: the CLI's --recommend path does NOT enforce its declared maxDuration=7200s. The deadline is only created in the non-recommend path at AutotuneCommand.swift:157-161. runAutotuneRecommend() at :131-139 returns before that and calls AutotuneRecommendationBenchmarker().benchmarks() with no deadline. So 7260s is not "CLI cap + grace" — it IS the outer bound of autotune runtime, chosen App-side. Value unchanged (7260s is still correct). Rewrite: - Rationale comment: honest description of App-side authoritative ceiling with 4 concrete reasons the value is 7260s. - Test invariants: pin to "realistic worst-case autotune runtime" (7200s = 10 candidates × 720s ceiling) NOT to a mirrored CLI cap that doesn't exist. Upper bound still 2.5h untenable-spinner guard. Tests: pass locally. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
antfleet-ops
approved these changes
Jul 5, 2026
antfleet-ops
left a comment
Collaborator
There was a problem hiding this comment.
Autotune timeout fix. 5 rounds of 3-lane codex audit; all lanes converged 0 C/H/M. LOWs documented. R2 CODE-M-2 orphan child correctly deferred to CLI-side follow-up. Approved.
3 tasks
Augustas11
added a commit
that referenced
this pull request
Jul 5, 2026
Cadence bump for smoke re-run of PR #396 (autotune process timeout 30s -> 7260s / 2h1m). v1.8.4 prerelease is the first .pkg containing the timeout fix. Grepped for '1.8.3' in both source + tests per [feedback-version-bump-grep-both-versions]. Historical 1.8.3 docstring references in AutotuneRecommendationRunner.swift and AutotuneRecommendationRunnerTimeoutTests.swift are intentional records of the smoke failure that motivated the fix and stay pinned to 1.8.3. coordinator.yaml latest_binary_version stays at 1.8.3 until v1.8.4 prerelease validation confirms the fix — follow-up PR bumps that. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Raise
AutotuneRecommendationRunner.processTimeoutfrom30sto7260s(2h1m) so fresh-install onboarding actually reaches.liveon real hardware.Root cause: The 30s value in
AutotuneRecommendationRunner.swiftwas ~240× too short. Every fresh-install v1.8.3 smoke on this Mac hitAutotuneRecommendationError.timedOutat T+30s while autotune was still spawning MLX probes. Autotune runs Stage-1 benchmarks against every non-blocked, RAM-eligible catalog row — each candidate takes up to 720s (readyTimeout 120s + prewarm probe 300s + measured probe 300s per SPEC-023 v1.7.5), catalog cardinality scales with Mac tier, and the CLI--recommendpath does NOT enforce its own declaredmaxDuration=7200s. So the App-side timeout IS the authoritative ceiling for autotune runtime.Value: 7260s = realistic worst-case autotune (10 candidates × 720s = 7200s) + 60s grace. Below 2.5h untenable-spinner ceiling. Median UX unaffected (empirical: 2-5 min).
Files
phase3-binary/app/Sources/Malibu/Onboarding/AutotuneRecommendationRunner.swift— bumpprocessTimeout30→7260s, visibilityprivate static→staticso tests can pin the invariant, rewrite rationale comment with 4 numbered reasons for the value and the honest note that the CLI--recommendpath doesn't enforce its ownmaxDuration.phase3-binary/app/Tests/MalibuTests/AutotuneRecommendationRunnerTimeoutTests.swift— new test file, 2 tests pinning: (1)processTimeout ≥ realisticWorstCaseAutotuneSec (7200s); (2)processTimeout ≤ untenableSpinnerCeilingSec (2.5h).specs/AUDIT_FIX_AUTOTUNE_TIMEOUT_{COMMON,CODE,SECURITY,ARCHITECT}_PROMPT.md— 3-lane codex audit prompts kept in-tree for future reference and R2-R5 convergence history.Audit convergence (5 rounds, R5 final)
probeOncestep--recommenditerates all non-blocked RAM-eligible rows; M-2: pre-existing CLI-side orphan-child bug (deferred, see below)--recommenddoesn't enforcemaxDurationAll CRITICAL / HIGH / MEDIUM findings converged to zero.
LOW findings — documented, deferred
Per project convention: LOWs ship with PR-body docs.
ProcessOutputBufferatAutotuneRecommendationRunner.swift:33-50is unbounded. Realistic autotune stdout is single-digit MB (per-probe JSON + final result). Would need a compromised/spoofed CLI to matter. Local DoS/OOM only.--recommendpath atphase3-binary/Sources/macprovider-cli/AutotuneCommand.swift:131-139does NOT enforce declaredmaxDuration=7200s(deadline is only installed in non-recommend at :157-161). CLI-side fix belongs in a follow-up PR.AutotuneRecommendationError.timedOutcopy is generic. At 2h1m, human failure copy per SPEC-026 §6.4 should convey the wait was long. Follow-up.Deferred to CLI-side follow-up
macprovider-cli servesubprocess when App-side timeout fires.Stage1Proberspawns theservechild atStage1Iterator.swift:436-442and relies ondefer { runner.stop(...) }(:443-445). The CLI--recommendpath does NOT install signal handlers (only non-recommend does atAutotuneCommand.swift:157-161), soprocess.terminate()from the App kills the parent CLI without giving Stage1Prober a cleanup path. The child leaks and can hold port 18080 + model memory. This is a pre-existing CLI-side bug that exists at any App-side timeout value ≥ 0; not introduced by this PR. Fix belongs inphase3-binary/Sources/macprovider-cli/AutotuneCommand.swiftto install signal handlers on the recommend path.Test plan
xcodebuild -project Malibu.xcodeproj -scheme MalibuTests test).liveend-to-end🤖 Generated with Claude Code