Repository navigation
Let Think models use their own default reasoning - #570
Draft
stevyhacker wants to merge 5 commits into
Draft
stevyhacker wants to merge 5 commits into
stevyhacker wants to merge 5 commits into
Conversation
Tasks no longer carry thinking budgets. A request either uses the model's default reasoning or turns it off; only instant tasks (Compose, selection actions, window handoff, autocomplete's chat continuation, the live radar pass, CLI/MCP short answers) turn it off. The Reasoning menu stays as one global override that no task lowers. Requests that may think get room for it on top of the answer (capped at 8,192), a reply that spends its whole output thinking is retried once with reasoning off, the built-in server runs with --reasoning auto so each chat template decides, and a notes job that may reason gets twice the time and output budget. Same on macOS and on Windows and Linux; ADR 0012 records why.
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
lokalbot | a03db0a | Commit Preview URL Branch Preview URL |
Nov 21 2026, 10:48 PM |
Qwen3.5 4B thinking off vs model default and Inkling 2 Mini, through the Windows and Linux notes code (new notes_replay example) on llama.cpp b12613.
llama-server asks every template that can think to think, under --reasoning on and auto alike, so a model-default request to Gemma 4 or the small Qwen models thought with no room for it; the runtime e2e notes for Qwen3.5 0.8B came back empty. On Automatic those models now get enable_thinking false, in the app and in Agent Mode.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Think tasks no longer carry per-task thinking budgets: each model uses its own default reasoning, and only instant tasks that someone waits on (dictation Compose, selection actions, window handoff, autocomplete's chat continuation, the live Meeting Radar pass, CLI/MCP short answers) turn thinking off. Most current models think by default, and the budgets (0 for notes, digests and personal learning; 128–1,024 elsewhere) kept them from working as trained while making every task a tuning point. The Reasoning menu stays as one global override; Automatic now means "the model decides" and a chosen level applies to every task that isn't Off.
What replaces the budgets, on macOS and on Windows/Linux alike:
TaskReasoning.modelDefaultor.off(RustTaskReasoning::{ModelDefault, Off})..offproduces exactly today's budget-0 request on every provider (thinking_budget_tokens: 0+enable_thinking: false, Ollamathink: false, OpenRoutereffort: none, lowest Claude effort).min(8,192, answer)(or the chosen level's budget) added tomax_tokens; on the built-in server the same number goes out asthinking_budget_tokensso an answer always arrives. This replaces the old "half of max_tokens" squeeze.complete_full/stream_full), and both attempts are charged to the meeting budget. Notes', digest's and Dream's own retries run Off so retries don't stack.enable_thinking = enable_reasoning != 0 && template supports it, under both--reasoning onandauto), so on Automatic LokalBot sendsenable_thinking: falseto models whose default is no thinking (Gemma 4, Qwen3.5 0.8B/2B, …), and Agent Mode does the same from the level file'sdefaultOff. A chosen level sendsenable_thinking: trueonly to those models. The main server now runs--reasoning auto(same per-request behaviour ason, without forcing a template default).Rationale is in ADR 0012 ("Reasoning and output budgets"), which this rewrites.
Validation
macOS (worktree on master 42374bd):
Unit tests were not run locally: the Mac's screen was locked, which stops the unit-test host from connecting. Relying on CI for the Swift unit suite.
Windows and Linux, from
LinuxWindows/(Ubuntu 24.04 box):Notes benchmark (report): the Windows/Linux notes code (
examples/notes_replay.rs) against llama.cpp b12613 on an RTX 3090, app server arguments, synthetic sample meeting (en, sr) and AMI ES2004a, one run each:No request ran out of room and the fallback never fired; both models thought up to the cap (the answer size) on nearly every request. Qwen's English key points became near-verbatim quotes with thinking on. The macOS notes code was not run (locked screen); it sends the same requests.
Changed test expectations
Every change below follows from one policy change: a task no longer lowers the model's reasoning, and Automatic no longer derives an effort from a task budget. The old expectations encoded the per-task budget policy this PR removes; none of them describe a requirement that still holds.
macOS:
TextEngineTests/testBuiltInHighReasoningBudgetReservesHalfOfBoundedOutput→testBuiltInModelDefaultGetsThinkingRoomOnTopOfTheAnswer: 8,192 max / 4,096 thinking → 16,384 max / 8,192 thinking. Halving squeezed the answer; room is now added on top.TextEngineTests/testBuiltInHighReasoningUsesFullCeilingWithoutOutputLimit→…WithoutOutputLimitCapsOnlyTheThinking: same 8,192 cap, now via the allowance.TextEngineTests/testExplicitReasoningBudgetCanDisableThinkingForOneRequest,testDisabledThinkingPreservesOtherTemplateArguments,testOllamaRequests…ZeroBudget: budget 0 →.off; bodies unchanged.TextEngineTests/testPositiveThinkingBudgetPreservesTemplatePolicy→testThinkingCapPreservesTemplatePolicy: thinking 128 with max 512 → thinking 128 with max 640 (room added on top).TextEngineTests/testGenericExternalEngineLeavesReasoningAtServerDefault: max_tokens 700 → 1,000 with a 300-token allowance; still no reasoning field.TextEngineTests/testRequestLevelIsTheChosenLevelCappedByTheTask…→…ClampedToTheModelUnlessTheTaskIsOff: High with a 256/1,024 budget gave Low/Medium → High; task budgets no longer exist.TextEngineTests/testGPT6ModelsOfferTheirDocumentedLevels: Max with a 1,024 budget gave "medium" → "max"; Automatic gave "medium" (derived from the budget) → no field (model default),max_completion_tokens900 → 1,800.TextEngineTests/testCerebrasRequestCarriesTheChosenEffortCappedByTheTask→…UnlessTheTaskIsOff: budget 256 gave "low" → the chosen "medium".TextEngineTests/testBuiltInReasoningLevelSetsTheThinkingBudget: High with max 2,000 gave 1,000 thinking → 8,192 thinking and 10,192 max.TextEngineTests/testOfficialOpenAIUsesModernReasoningFieldsWithoutSamplingExtension: effort "low" from a 256 budget → no field;max_completion_tokens700 → 1,400.TextEngineTests/testOpenRouterUsesNativeReasoningAndPrivacyRouting→testOpenRouterLeavesReasoningAtTheModelDefaultAndRoutesPrivately:reasoning.max_tokens256 → no reasoning object; max 700 → 1,400; temperature now sent.TextEngineTests/testOpenRouterPlainChatKeepsTemperatureWithoutForcingParameters:max_tokens300 → 600. The model's default is unknown, so it may reason and gets room for it on top of the answer; temperature and the absence ofrequire_parametersare unchanged.TextEngineTests/testOpenRouterParameterMismatch…,testOpenRouterMandatoryReasoningError…: the predicate now keys onrequestsReasoningOff; positive-budget cases no longer exist.AnthropicEngineTests/testEffortFollowsTaskBudgetAndChosenLevel→testEffortFollowsTheChosenLevelOrTheModelDefaultUnlessTheTaskIsOff: budget 4,096 gave Medium → model default (nil).ClaudePlanEngineTests/testChosenEffortIsCappedByTheTaskBudget→testChosenEffortAppliesToEveryTaskThatIsNotOff: Max with budget 4,096 gave Medium → Max.AntigravityPlanEngineTests/testEffortAvoidsConflict…CapsByBudget→…DropsToTheLowestForOffTasks: XHigh with budget 4,096 gave Medium → XHigh.ChatGPTPlanTests/testGPT6RequestsCarryTheChosenLevel: renamed budget argument; Off still gives "low"/"none".DayDigestEvidenceTests(focus/aggregation calls): budget 256/512 → model default; retries budget 0 →.off.DreamLoopTests/testDreamBoundsReasoningAndOutput→testDreamBoundsOutputAndLeavesReasoningToTheModel: budget 1,024 → model default.MeetingNotesGeneratorTests/testOnePassProducesSummaryActionsDecisionsAndQuestions: notes budget 0 → model default.AgentLLMEndpointTests/testMainLLMUsesHighReasoningAnd32KContextPolicy→testMainLLMRunsWithAutomaticReasoningAndA32KContext:--reasoning on→auto.ModelServerScenarioTests/testReasoningHeavyModelFitsOnTheLargerRetry→testReasoningHeavyModelFitsInTheRoomLeftForThinking: two focus requests (2,048, 4,096) → one (4,096), because the first request now has room to think.Windows and Linux:
think.rs:thinking_budget_is_capped_at_half_the_outputreplaced;dialects_get_their_own_fields(no budget-derived effort, no OpenRouter{max_tokens, exclude}, max includes the allowance);gpt_6_models_offer_their_documented_levels(budget 1,024 gave "medium" → Automatic sends nothing, High sends "high");ollama_uses_api_chat_with_schema_and_think(num_predictincludes the allowance).anthropic.rsmodel_traits_follow_anthropics_documentation: Max with budget 4,096 gave Medium → Max.responses_requests_stream_without_storage_or_sampling"low" → "high";arguments_carry_the_mac_isolation_flags"medium" → "max" (Off gives "low");arguments_keep_content_and_schema_off_argv_and_avoid_effort_conflict"low" → "high".digest/tests.rstwo_passes_build_tasks_and_follow_ups: max 2,048/thinking 256 → 4,096/2,048.follow_up.rsdrafts_from_the_fixture_meeting: max 900 → 1,800, thinking 900.notes/tests.rsone_pass_renders_notes_outcomes_and_header:thinking_budget_tokens0 → absent,enable_thinking: falsekept. Notes are model default now, and qwen3.5-0.8b's default is no thinking, which llama-server only honours when told.runtime.rsports_and_arguments_follow_the_roles:--reasoning on→auto.{level, defaultOff}.Risk / rollout notes
auto.ModelCatalog.Entry.disablesThinkingstays as a decode-only legacy field.