Skip to content

fix(llm): honor SIMARD_LLM_PROVIDER / AMPLIHACK_LLM_PROVIDER env override#4477

Merged
rysweet merged 1 commit into
mainfrom
fix/llm-provider-env-override
Apr 25, 2026
Merged

fix(llm): honor SIMARD_LLM_PROVIDER / AMPLIHACK_LLM_PROVIDER env override#4477
rysweet merged 1 commit into
mainfrom
fix/llm-provider-env-override

Conversation

@rysweet

@rysweet rysweet commented Apr 25, 2026

Copy link
Copy Markdown
Owner

Why

Simard's OODA daemon was scoring 0.00% on every L1-L12 evaluation while reporting ✓ completed — a textbook hollow-success bug. Root cause: amplihack's LLM router defaults to the bundled Claude Code CLI, which is "Not logged in" by default. The daemon's SIMARD_LLM_PROVIDER=copilot env var was being ignored.

Two intertwined bugs.

Bug 1: _detect_launcher is purely file-based

amplihack.llm.client._detect_launcher reads <project_root>/.claude/runtime/launcher_context.json and returns "claude" when the file is absent, stale, or malformed. Embedded callers that don't go through amplihack copilot — Simard's Rust daemon imports amplihack.eval directly via Python subprocess — never have a launcher context written, so detection unconditionally falls through to "claude". The SIMARD_LLM_PROVIDER env var is never read.

Bug 2: copilot SDK probe imports a non-existent module

from copilot.types import MessageOptions, SessionConfig  # ModuleNotFoundError

copilot.types doesn't exist in copilot >= 0.1.0. That ImportError silently set _COPILOT_SDK_OK = False, making the entire copilot path dead code. _query_copilot was also calling a stale API (client.start(), session.send_and_wait(MessageOptions(prompt=...))) that no longer exists.

Fix

Selector (_detect_launcher):

  1. Explicit env override (AMPLIHACK_LLM_PROVIDER, then SIMARD_LLM_PROVIDER) — wins unconditionally
  2. File-based detection via LauncherDetector
  3. Default "claude"

Recognized provider values: claude/anthropic/claude-code and copilot/github-copilot/gh-copilot/rustyclawd. Unknown values fall through to file detection.

Dispatch (completion): when an env override is present, refuse to silently fall back across providers. If the user pinned copilot but the SDK is unavailable, log a warning and return "" rather than masking the misconfiguration by routing to Claude.

Copilot SDK (_query_copilot): rewritten against the real copilot 0.1.0 API:

async with CopilotClient() as client:
    session = await client.create_session(
        on_permission_request=PermissionHandler.approve_all,
        working_directory=str(project_root),
    )
    event = await session.send_and_wait(prompt, timeout=float(QUERY_TIMEOUT))

Verification

End-to-end smoke against the installed copilot SDK:

$ SIMARD_LLM_PROVIDER=copilot python -c "
import asyncio; from amplihack.llm import completion
print(asyncio.run(completion(messages=[{'role':'user','content':'Reply with just the word: OK'}])))
"
OK

Tests

17 new tests in tests/llm/test_provider_env_override.py:

  • Value normalization (case, whitespace, aliases)
  • AMPLIHACK_LLM_PROVIDER priority over SIMARD_LLM_PROVIDER
  • Env override beats existing launcher_context.json on disk
  • Unknown values fall through to file detection
  • No-silent-fallback contract: pinning copilot must NOT route to Claude if copilot SDK is unavailable, and vice versa

All 19 LLM-related tests pass (17 new + 2 existing in tests/eval/test_llm_grader.py).

Impact

Once merged and deployed via uv tool upgrade amplihack, Simard's OODA cycles should produce real, non-zero L1-L12 scores instead of the current 0.00% hollow-success regime. Further, any caller that sets SIMARD_LLM_PROVIDER=copilot or AMPLIHACK_LLM_PROVIDER=copilot in the environment now gets the behavior that name advertises, with no .claude/runtime/launcher_context.json ritual required.

…ride

Two related bugs were causing every L1-L12 evaluation in Simard's OODA
daemon to silently route to the bundled Claude Code CLI — which is "Not
logged in" by default — and produce empty completions that
metacognition_grader swallowed via its JSON-parse-fail path. Result: every
test scored 0.00% but the daemon reported "completed", a textbook hollow
success.

Bug #1: amplihack.llm.client._detect_launcher was purely file-based. It
read <project_root>/.claude/runtime/launcher_context.json and returned
"claude" if the file was missing, stale, or malformed. Embedded callers
that don't go through `amplihack copilot` (Simard's Rust daemon imports
amplihack.eval directly via Python subprocess) never had a launcher
context written, so detection unconditionally fell through to "claude" —
even though SIMARD_LLM_PROVIDER=copilot was set in the daemon env.

Bug #2: the copilot SDK probe imported a module that doesn't exist in
copilot >= 0.1.0:
    from copilot.types import MessageOptions, SessionConfig
That ImportError silently set _COPILOT_SDK_OK = False, so the copilot
path was effectively dead code regardless of which launcher was selected.
_query_copilot was also calling a stale API (client.start(),
session.send_and_wait(MessageOptions(prompt=...))) that no longer exists.

Fix:
  * _detect_launcher() now consults AMPLIHACK_LLM_PROVIDER and
    SIMARD_LLM_PROVIDER first; an explicit override wins unconditionally.
    File-based detection is the second tier; "claude" remains the final
    fallback.
  * Recognized provider values include claude/anthropic/claude-code and
    copilot/github-copilot/gh-copilot/rustyclawd. Unknown values fall
    through to file detection.
  * completion() now refuses to silently fall back across providers when
    an env override is present. If the user pinned copilot but the SDK
    is unavailable, we log a warning and return "" rather than masking
    the misconfiguration by routing to Claude.
  * _query_copilot() rewritten to use the real copilot 0.1.0 API:
    `async with CopilotClient() as client: session = await
    client.create_session(on_permission_request=PermissionHandler.approve_all,
    working_directory=...); event = await session.send_and_wait(prompt,
    timeout=...)`. Drops the bogus copilot.types import.

End-to-end smoke test against the installed copilot SDK confirms the
override path now produces real completions:

    SIMARD_LLM_PROVIDER=copilot python -c \
      "import asyncio; from amplihack.llm import completion; \
       print(asyncio.run(completion(messages=[{'role':'user',
       'content':'Reply with just the word: OK'}])))"
    OK

Tests: 17 new tests in tests/llm/test_provider_env_override.py covering
provider value normalization, env priority, file-vs-env precedence, and
the no-silent-fallback contract. All 19 LLM-related tests pass.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant