fix(llm): honor SIMARD_LLM_PROVIDER / AMPLIHACK_LLM_PROVIDER env override#4477
Merged
Conversation
…ride Two related bugs were causing every L1-L12 evaluation in Simard's OODA daemon to silently route to the bundled Claude Code CLI — which is "Not logged in" by default — and produce empty completions that metacognition_grader swallowed via its JSON-parse-fail path. Result: every test scored 0.00% but the daemon reported "completed", a textbook hollow success. Bug #1: amplihack.llm.client._detect_launcher was purely file-based. It read <project_root>/.claude/runtime/launcher_context.json and returned "claude" if the file was missing, stale, or malformed. Embedded callers that don't go through `amplihack copilot` (Simard's Rust daemon imports amplihack.eval directly via Python subprocess) never had a launcher context written, so detection unconditionally fell through to "claude" — even though SIMARD_LLM_PROVIDER=copilot was set in the daemon env. Bug #2: the copilot SDK probe imported a module that doesn't exist in copilot >= 0.1.0: from copilot.types import MessageOptions, SessionConfig That ImportError silently set _COPILOT_SDK_OK = False, so the copilot path was effectively dead code regardless of which launcher was selected. _query_copilot was also calling a stale API (client.start(), session.send_and_wait(MessageOptions(prompt=...))) that no longer exists. Fix: * _detect_launcher() now consults AMPLIHACK_LLM_PROVIDER and SIMARD_LLM_PROVIDER first; an explicit override wins unconditionally. File-based detection is the second tier; "claude" remains the final fallback. * Recognized provider values include claude/anthropic/claude-code and copilot/github-copilot/gh-copilot/rustyclawd. Unknown values fall through to file detection. * completion() now refuses to silently fall back across providers when an env override is present. If the user pinned copilot but the SDK is unavailable, we log a warning and return "" rather than masking the misconfiguration by routing to Claude. * _query_copilot() rewritten to use the real copilot 0.1.0 API: `async with CopilotClient() as client: session = await client.create_session(on_permission_request=PermissionHandler.approve_all, working_directory=...); event = await session.send_and_wait(prompt, timeout=...)`. Drops the bogus copilot.types import. End-to-end smoke test against the installed copilot SDK confirms the override path now produces real completions: SIMARD_LLM_PROVIDER=copilot python -c \ "import asyncio; from amplihack.llm import completion; \ print(asyncio.run(completion(messages=[{'role':'user', 'content':'Reply with just the word: OK'}])))" OK Tests: 17 new tests in tests/llm/test_provider_env_override.py covering provider value normalization, env priority, file-vs-env precedence, and the no-silent-fallback contract. All 19 LLM-related tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced Apr 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Simard's OODA daemon was scoring 0.00% on every L1-L12 evaluation while reporting
✓ completed— a textbook hollow-success bug. Root cause: amplihack's LLM router defaults to the bundled Claude Code CLI, which is "Not logged in" by default. The daemon'sSIMARD_LLM_PROVIDER=copilotenv var was being ignored.Two intertwined bugs.
Bug 1:
_detect_launcheris purely file-basedamplihack.llm.client._detect_launcherreads<project_root>/.claude/runtime/launcher_context.jsonand returns"claude"when the file is absent, stale, or malformed. Embedded callers that don't go throughamplihack copilot— Simard's Rust daemon importsamplihack.evaldirectly via Python subprocess — never have a launcher context written, so detection unconditionally falls through to"claude". TheSIMARD_LLM_PROVIDERenv var is never read.Bug 2: copilot SDK probe imports a non-existent module
copilot.typesdoesn't exist incopilot >= 0.1.0. ThatImportErrorsilently set_COPILOT_SDK_OK = False, making the entire copilot path dead code._query_copilotwas also calling a stale API (client.start(),session.send_and_wait(MessageOptions(prompt=...))) that no longer exists.Fix
Selector (
_detect_launcher):AMPLIHACK_LLM_PROVIDER, thenSIMARD_LLM_PROVIDER) — wins unconditionallyLauncherDetector"claude"Recognized provider values:
claude/anthropic/claude-codeandcopilot/github-copilot/gh-copilot/rustyclawd. Unknown values fall through to file detection.Dispatch (
completion): when an env override is present, refuse to silently fall back across providers. If the user pinnedcopilotbut the SDK is unavailable, log a warning and return""rather than masking the misconfiguration by routing to Claude.Copilot SDK (
_query_copilot): rewritten against the realcopilot 0.1.0API:Verification
End-to-end smoke against the installed
copilotSDK:Tests
17 new tests in
tests/llm/test_provider_env_override.py:AMPLIHACK_LLM_PROVIDERpriority overSIMARD_LLM_PROVIDERlauncher_context.jsonon diskAll 19 LLM-related tests pass (17 new + 2 existing in
tests/eval/test_llm_grader.py).Impact
Once merged and deployed via
uv tool upgrade amplihack, Simard's OODA cycles should produce real, non-zero L1-L12 scores instead of the current 0.00% hollow-success regime. Further, any caller that setsSIMARD_LLM_PROVIDER=copilotorAMPLIHACK_LLM_PROVIDER=copilotin the environment now gets the behavior that name advertises, with no.claude/runtime/launcher_context.jsonritual required.