Skip to content

Let Think models use their own default reasoning - #570

Draft
stevyhacker wants to merge 5 commits into
masterfrom
claude/reasoning-model-defaults
Draft

stevyhacker wants to merge 5 commits into
masterfrom
claude/reasoning-model-defaults

Conversation

@stevyhacker

@stevyhacker stevyhacker commented Nov 21, 2026 •

Copy link
Copy Markdown
Owner

Summary

Think tasks no longer carry per-task thinking budgets: each model uses its own default reasoning, and only instant tasks that someone waits on (dictation Compose, selection actions, window handoff, autocomplete's chat continuation, the live Meeting Radar pass, CLI/MCP short answers) turn thinking off. Most current models think by default, and the budgets (0 for notes, digests and personal learning; 128–1,024 elsewhere) kept them from working as trained while making every task a tuning point. The Reasoning menu stays as one global override; Automatic now means "the model decides" and a chosen level applies to every task that isn't Off.

What replaces the budgets, on macOS and on Windows/Linux alike:

  • One intent per request: TaskReasoning.modelDefault or .off (Rust TaskReasoning::{ModelDefault, Off}). .off produces exactly today's budget-0 request on every provider (thinking_budget_tokens: 0 + enable_thinking: false, Ollama think: false, OpenRouter effort: none, lowest Claude effort).
  • Known defaults for built-in models (from each chat template): Qwen3.5 4B and Qwen3.6 think; Qwen3.5 0.8B/2B, Gemma 4, LFM2.5 1.2B, Granite, Ministral don't; LFM2.5 2.6B always reasons.
  • Room to think on top of the answer: a request that may reason gets min(8,192, answer) (or the chosen level's budget) added to max_tokens; on the built-in server the same number goes out as thinking_budget_tokens so an answer always arrives. This replaces the old "half of max_tokens" squeeze.
  • One fallback: a reply that stopped at its limit with no visible answer is retried once with reasoning off (OpenAI-compatible, Ollama, Anthropic; Rust complete_full/stream_full), and both attempts are charged to the meeting budget. Notes', digest's and Dream's own retries run Off so retries don't stack.
  • Models that don't think by default are told not to: llama-server asks every thinking-capable template to think (enable_thinking = enable_reasoning != 0 && template supports it, under both --reasoning on and auto), so on Automatic LokalBot sends enable_thinking: false to models whose default is no thinking (Gemma 4, Qwen3.5 0.8B/2B, …), and Agent Mode does the same from the level file's defaultOff. A chosen level sends enable_thinking: true only to those models. The main server now runs --reasoning auto (same per-request behaviour as on, without forcing a template default).
  • Notes budget: a notes job that may reason gets twice the time and output tokens and halves each part's share, so answer sizes stay where they were; a request whose thinking room alone won't fit the context goes Off.

Rationale is in ADR 0012 ("Reasoning and output budgets"), which this rewrites.

Validation

macOS (worktree on master 42374bd):

xcodegen generate
xcodebuild … -scheme LokalBot -configuration Debug CODE_SIGNING_ALLOWED=NO -only-testing:LokalBotTests build-for-testing
# ** TEST BUILD SUCCEEDED **
swiftlint lint --strict --quiet <changed Swift files>   # exit 0
python3 Scripts/ci/comment-ratchet.py --check            # Comment lines: 0 (baseline 0)

Unit tests were not run locally: the Mac's screen was locked, which stops the unit-test host from connecting. Relying on CI for the Swift unit suite.

Windows and Linux, from LinuxWindows/ (Ubuntu 24.04 box):

cargo fmt --all --check                                   # pass
cargo clippy --workspace --all-targets -- -D warnings     # pass
cargo test -p lokalbot-ai -p lokalbot-core -p lokalbot-agent -p lokalbot-engine   # pass (lokalbot-ai 565, 3 ignored)
cargo test -p lokalbot-i18n -p lokalbot-app -p lokalbot-cli                       # pass
bash crates/lokalbot-agent/resources/tests/run-pi-runtime-tests.sh                # 44 pass, 1 skip
bash Scripts/tests/run-pi-runtime-tests.sh                                        # 46 pass
python3 scripts/sync-localizations.py --check                                     # OK
cargo test -p lokalbot-ai --test e2e_runtime -- --ignored --exact installs_runtime_downloads_models_and_answers   # pass
cargo test -p lokalbot-ai --test e2e_runtime -- --ignored --exact generates_notes_for_a_fixture_meeting           # pass (23.7 s); failed before f079ff2c, see "Models that don't think by default are told not to"

Notes benchmark (report): the Windows/Linux notes code (examples/notes_replay.rs) against llama.cpp b12613 on an RTX 3090, app server arguments, synthetic sample meeting (en, sr) and AMI ES2004a, one run each:

Sample en Sample sr AMI ES2004a Facts en / sr
Qwen3.5 4B, thinking off (master) 6.4 s 5.0 s 11.5 s 9/14 · 3/14 (notes in English)
Qwen3.5 4B, model default (this PR) 32.2 s 33.1 s 70.1 s 11/14 · 8/14
Inkling 2 Mini, model default 120.6 s 114.3 s 170.0 s 11/14 · 13/14

No request ran out of room and the fallback never fired; both models thought up to the cap (the answer size) on nearly every request. Qwen's English key points became near-verbatim quotes with thinking on. The macOS notes code was not run (locked screen); it sends the same requests.

Changed test expectations

Every change below follows from one policy change: a task no longer lowers the model's reasoning, and Automatic no longer derives an effort from a task budget. The old expectations encoded the per-task budget policy this PR removes; none of them describe a requirement that still holds.

macOS:

  • TextEngineTests/testBuiltInHighReasoningBudgetReservesHalfOfBoundedOutput → testBuiltInModelDefaultGetsThinkingRoomOnTopOfTheAnswer: 8,192 max / 4,096 thinking → 16,384 max / 8,192 thinking. Halving squeezed the answer; room is now added on top.
  • TextEngineTests/testBuiltInHighReasoningUsesFullCeilingWithoutOutputLimit → …WithoutOutputLimitCapsOnlyTheThinking: same 8,192 cap, now via the allowance.
  • TextEngineTests/testExplicitReasoningBudgetCanDisableThinkingForOneRequest, testDisabledThinkingPreservesOtherTemplateArguments, testOllamaRequests…ZeroBudget: budget 0 → .off; bodies unchanged.
  • TextEngineTests/testPositiveThinkingBudgetPreservesTemplatePolicy → testThinkingCapPreservesTemplatePolicy: thinking 128 with max 512 → thinking 128 with max 640 (room added on top).
  • TextEngineTests/testGenericExternalEngineLeavesReasoningAtServerDefault: max_tokens 700 → 1,000 with a 300-token allowance; still no reasoning field.
  • TextEngineTests/testRequestLevelIsTheChosenLevelCappedByTheTask… → …ClampedToTheModelUnlessTheTaskIsOff: High with a 256/1,024 budget gave Low/Medium → High; task budgets no longer exist.
  • TextEngineTests/testGPT6ModelsOfferTheirDocumentedLevels: Max with a 1,024 budget gave "medium" → "max"; Automatic gave "medium" (derived from the budget) → no field (model default), max_completion_tokens 900 → 1,800.
  • TextEngineTests/testCerebrasRequestCarriesTheChosenEffortCappedByTheTask → …UnlessTheTaskIsOff: budget 256 gave "low" → the chosen "medium".
  • TextEngineTests/testBuiltInReasoningLevelSetsTheThinkingBudget: High with max 2,000 gave 1,000 thinking → 8,192 thinking and 10,192 max.
  • TextEngineTests/testOfficialOpenAIUsesModernReasoningFieldsWithoutSamplingExtension: effort "low" from a 256 budget → no field; max_completion_tokens 700 → 1,400.
  • TextEngineTests/testOpenRouterUsesNativeReasoningAndPrivacyRouting → testOpenRouterLeavesReasoningAtTheModelDefaultAndRoutesPrivately: reasoning.max_tokens 256 → no reasoning object; max 700 → 1,400; temperature now sent.
  • TextEngineTests/testOpenRouterPlainChatKeepsTemperatureWithoutForcingParameters: max_tokens 300 → 600. The model's default is unknown, so it may reason and gets room for it on top of the answer; temperature and the absence of require_parameters are unchanged.
  • TextEngineTests/testOpenRouterParameterMismatch…, testOpenRouterMandatoryReasoningError…: the predicate now keys on requestsReasoningOff; positive-budget cases no longer exist.
  • AnthropicEngineTests/testEffortFollowsTaskBudgetAndChosenLevel → testEffortFollowsTheChosenLevelOrTheModelDefaultUnlessTheTaskIsOff: budget 4,096 gave Medium → model default (nil).
  • ClaudePlanEngineTests/testChosenEffortIsCappedByTheTaskBudget → testChosenEffortAppliesToEveryTaskThatIsNotOff: Max with budget 4,096 gave Medium → Max.
  • AntigravityPlanEngineTests/testEffortAvoidsConflict…CapsByBudget → …DropsToTheLowestForOffTasks: XHigh with budget 4,096 gave Medium → XHigh.
  • ChatGPTPlanTests/testGPT6RequestsCarryTheChosenLevel: renamed budget argument; Off still gives "low"/"none".
  • DayDigestEvidenceTests (focus/aggregation calls): budget 256/512 → model default; retries budget 0 → .off.
  • DreamLoopTests/testDreamBoundsReasoningAndOutput → testDreamBoundsOutputAndLeavesReasoningToTheModel: budget 1,024 → model default.
  • MeetingNotesGeneratorTests/testOnePassProducesSummaryActionsDecisionsAndQuestions: notes budget 0 → model default.
  • AgentLLMEndpointTests/testMainLLMUsesHighReasoningAnd32KContextPolicy → testMainLLMRunsWithAutomaticReasoningAndA32KContext: --reasoning on → auto.
  • ModelServerScenarioTests/testReasoningHeavyModelFitsOnTheLargerRetry → testReasoningHeavyModelFitsInTheRoomLeftForThinking: two focus requests (2,048, 4,096) → one (4,096), because the first request now has room to think.

Windows and Linux:

  • think.rs: thinking_budget_is_capped_at_half_the_output replaced; dialects_get_their_own_fields (no budget-derived effort, no OpenRouter {max_tokens, exclude}, max includes the allowance); gpt_6_models_offer_their_documented_levels (budget 1,024 gave "medium" → Automatic sends nothing, High sends "high"); ollama_uses_api_chat_with_schema_and_think (num_predict includes the allowance).
  • anthropic.rs model_traits_follow_anthropics_documentation: Max with budget 4,096 gave Medium → Max.
  • Plans: responses_requests_stream_without_storage_or_sampling "low" → "high"; arguments_carry_the_mac_isolation_flags "medium" → "max" (Off gives "low"); arguments_keep_content_and_schema_off_argv_and_avoid_effort_conflict "low" → "high".
  • digest/tests.rs two_passes_build_tasks_and_follow_ups: max 2,048/thinking 256 → 4,096/2,048. follow_up.rs drafts_from_the_fixture_meeting: max 900 → 1,800, thinking 900.
  • notes/tests.rs one_pass_renders_notes_outcomes_and_header: thinking_budget_tokens 0 → absent, enable_thinking: false kept. Notes are model default now, and qwen3.5-0.8b's default is no thinking, which llama-server only honours when told.
  • runtime.rs ports_and_arguments_follow_the_roles: --reasoning on → auto.
  • pi extension (both suites): the level file is read as {level, defaultOff}.

Risk / rollout notes

  • Slower notes on the default model. Built-in Qwen3.5 4B now thinks on notes, digests, Dream, personal learning and Ask. The benchmark below will put numbers on it.
  • Remote providers on Automatic go back to their own defaults: Cerebras Qwen (High), GPT-5.2+/6 (Medium), Claude API / Claude plan / Antigravity (from Low via the old ceiling to the model default). More time and cost on paid plans; the allowance and the fallback keep notes from coming back empty.
  • Agent Mode shares the Think server: on Automatic, Gemma 4 stops being forced to think once the server runs auto.
  • No settings or schema migration. ModelCatalog.Entry.disablesThinking stays as a decode-only legacy field.

Tasks no longer carry thinking budgets. A request either uses the model's
default reasoning or turns it off; only instant tasks (Compose, selection
actions, window handoff, autocomplete's chat continuation, the live radar
pass, CLI/MCP short answers) turn it off. The Reasoning menu stays as one
global override that no task lowers.

Requests that may think get room for it on top of the answer (capped at
8,192), a reply that spends its whole output thinking is retried once with
reasoning off, the built-in server runs with --reasoning auto so each chat
template decides, and a notes job that may reason gets twice the time and
output budget. Same on macOS and on Windows and Linux; ADR 0012 records why.
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
lokalbot a03db0a Commit Preview URL

Branch Preview URL
Nov 21 2026, 10:48 PM

Qwen3.5 4B thinking off vs model default and Inkling 2 Mini, through the Windows and Linux notes code (new notes_replay example) on llama.cpp b12613.
llama-server asks every template that can think to think, under --reasoning on and auto alike, so a model-default request to Gemma 4 or the small Qwen models thought with no room for it; the runtime e2e notes for Qwen3.5 0.8B came back empty. On Automatic those models now get enable_thinking false, in the app and in Agent Mode.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant