BYOK: a model-name typo is indistinguishable from an absent key at startup

[jasperblues]

Problem

In setup-required mode a typo in a model name is indistinguishable from a key that has not arrived yet, so a deployment awaiting a key gets no startup validation of its model names at all.

Since #1888 / #1889, ConfigurableModelProvider treats an unresolvable model name in embabel.models as fatal at startup, except when default-llm resolved to a PlaceholderLlmService — the deployment stating that keys arrive at runtime. That exception is necessary: a pure BYOK deployment has no models registered, so every name it configures is unresolvable and a fatal check would make it unbootable.

The cost is that inside that exception, these two are the same to the platform:

embabel:
  models:
    default-llm: setup-required
    llms:
      best: gpt-4.1          # correct; will resolve once a key arrives
      cheapest: gpt-4.1-nanoo # typo; will NEVER resolve

Both names are simply unregistered at boot. Both warn. Both resolve to the placeholder. The first starts working when a user supplies a key; the second never does, and the failure surfaces as "no LLM is configured" — which points the operator at the key, the one thing that is not wrong.

So fail-fast came back for keyed deployments and did not come back for BYOK ones. That asymmetry is inherent to the current design rather than an oversight in it: at boot, with nothing registered, there is no information that separates the two.

Why it isn't trivially fixable

Distinguishing them means validating configured names against a catalogue of model names known to the framework, independently of what is registered.

There is no such catalogue. com.embabel.agent.api.models has twelve per-provider objects — OpenAiModels, AnthropicModels, DeepSeekModels, GoogleGenAiModels, MistralAiModels, OllamaModels, LmStudioModels, DockerLocalModels, DashScopeModels, MiniMaxModels, OciGenAiModels, ZaiModels — and nothing aggregates them.

Options

1. Aggregate catalogue + warn on unknown names. Build the union, and in setup-required mode warn specifically when a configured name matches nothing known. Catches gpt-4.1-nanoo. Cannot be fatal: a legitimately new model released after the framework version would trip it, and failing to boot on that would be worse than the problem.
2. Validate on first key. When a key arrives and models register, re-check the configured names and warn about any that are still unresolvable. Later than startup, but by then the information genuinely exists, and it is exact rather than heuristic.
3. Accept it, and document it. The failure is recoverable and the message can be improved — for example, NoLlmConfiguredException could name the model the role wanted, so an operator who has supplied a key sees cheapest wanted 'gpt-4.1-nanoo' rather than a bare "configure a key".

(2) and (3) compose well and neither needs a catalogue. (1) is the only one that catches the typo before a key ever arrives, and it is the only one that needs new shared infrastructure.

Suggested scope

(3) first, since it is small and makes the existing failure self-explanatory, then (2). (1) only if a catalogue turns out to be wanted for other reasons — it is a lot of surface area to add for this alone, and it dates with every model release.

Context

• Add an embabel-agent-starter-byok, so a BYOK app can boot without keys #1888 introduced the placeholder and setup-required mode
• LLM roles should resolve through an SPI to a materialized LlmService, not a static role→model map #1889 applies the same rule to role resolution
• ConfigurableModelProvider.reportUnsatisfiableRole is where the two cases currently converge