feat(pricing): add OpenRouter as second fallback pricing source - #1225
feat(pricing): add OpenRouter as second fallback pricing source#1225godlockin wants to merge 17 commits into
Conversation
roborev: Combined Review (
|
roborev: Combined Review (
|
The seed-and-refresh loop in cmd/agentsview previously only talked to the LiteLLM pricing catalog. When the upstream fetch failed (offline, DNS broken, rate-limited) and the embedded fallback snapshot did not contain the user's model, daily usage cost silently dropped to $0 — the same symptom that the fork model custom-pricing test guards against. Wire in OpenRouter's public /models endpoint as a second background source. LiteLLM stays first because it covers the public models agentsview normally parses; OpenRouter fills in fork-tuned and private model prices LiteLLM has not yet picked up. Each fetch failure is logged but never aborts the loop, so a partial outage of one upstream does not prevent the other from seeding. All successful results are merged with first-non-zero precedence per model_pattern. Adds: - internal/pricing/catalog/openrouter.go: fetcher and parser - internal/pricing/openrouter_test.go: unit tests for parser filtering, per-token-to-per-million conversion, and merge precedence - DefaultPricingSources() and MergePricing() in litellm.go exposing the source list and the merge helper so other callers (CLI statusline, future config-driven sources) can reuse them - refreshPricingFromSources() in cmd/agentsview/usage.go replacing the previous single-source call Also bumps the default-port assertion in cmd/agentsview/main_test.go and pg_test.go from 8080 to 9765 to match the port change landed earlier.
…aths
Add two regression tests around GetDailyUsage so future changes
that silently drop unpriced models can be caught immediately.
TestGetDailyUsageForkModelPricing inserts a custom model pattern
("internal-private-model") via UpsertModelPricing and verifies
that the day entry exists with the expected input/output token
counts and a non-zero cost. This is the case the user hit in
June 2026: a downstream fork that uses internal/private model
identifiers saw $0.00 cost because the model name did not
canonicalize to any upstream LiteLLM catalog key, even though
the database had rows with valid token_usage payloads.
TestGetDailyUsageUnknownModelHasZeroCostButCountsTokens covers
the fall-through case where a model is genuinely not priced:
the day entry must still exist (tokens are still counted), but
cost is $0. If this test ever fails with len(Daily)==0 the
upstream time-window SQL from issue kenn-io#904 has regressed.
seedPricing already kicks off one LiteLLM + OpenRouter fetch at startup and reapplies custom_model_pricing on top. But newly released or repriced upstream models were never picked up without a restart, which meant the dashboard could show stale rates for weeks between agentsview upgrades. periodicPricingRefresh runs a ticker in the server goroutine for the whole lifetime of the process. Every tick it reruns refreshPricingFromSources (LiteLLM merged with OpenRouter) and reapplies cfg.CustomModelPricing so a newly-published upstream rate cannot silently shadow the user's own override for fork/private model names. The loop is a no-op when interval <= 0 and unwinds on context cancel. Interval defaults to 24h — long enough to be gentle on the upstream catalogs, short enough that a mid-week price drop is picked up the next day.
The previous filter required strict text->text modality, which dropped multimodal-input, text-output models like MiniMax-M3 (text+image+video->text) and kimi-k2.5 (text+image->text) — the very models users reach via bare names such as `MiniMax-M3`. They still bill prompt/completion in text tokens, so filter on the output side only. Combined with the unqualified-suffix alias, sessions that log bare model names now resolve pricing directly from OpenRouter's public catalog.
OpenRouter ids are provider-qualified (`minimax/minimax-m3`), but agentsview sessions frequently record bare model names (`MiniMax-M3`, `kimi-k2.5`). The canonical resolver refused those lookups because every candidate key had a provider prefix and the same rank, so multiple providers tied and the row stayed unpriced. When a bare suffix is unique across the OpenRouter catalog, also emit an unqualified ModelPricing row so the resolver can rank it at the unqualified tier and resolve a bare user-side model. Shared suffixes (two providers publishing `kimi-k2.5`) still only produce prefixed rows to avoid fabricating OpenRouter-internal ambiguity.
cmd/agentsview/seedPricing writes the LiteLLM fallback snapshot to model_pricing and fires a background multi-source refresh, but it never wired the config-driven [custom_model_pricing] map into the running DB. The CLI statusline and pg serve paths call applyCustomPricing explicitly, which masked the gap, but the embedded HTTP server left db.customPricing at its zero value and silently priced fork-private models at \$0. Call applyCustomPricing immediately after seedPricing so fork owners can configure their internal-model rates in config.toml and have them flow into every GetDailyUsage call without a CLI detour. The read path was already correct: loadPricingMap merges db.customPricing on top of whatever model_pricing returns, so this fix is purely about plumbing the writer.
Cherry-pick bookkeeping for the upcoming upstream PR: - main_test.go, pg_test.go: restore 8080 default (the c876288 fork commit bundled in a 9765 port change that is not part of this PR) - usage_test.go: drop the TestRefreshPricingIfStale_* / TestEnsurePricingWithFetcher* tests that reference fork-local helpers (refreshPricingIfStale, ensurePricingWithFetcher) that do not exist on upstream; the upstream equivalent lives in the pricingrefresh package and is tested there - usage.go: add a local upsertPricing helper used by seedFallbackPricing and refreshPricingFromSources. The same helper exists inside internal/pricingrefresh on upstream but is unexported, so we duplicate the few lines here to keep the PR self-contained - openrouter.go, litellm.go: gofmt -s alignment cleanup - main.go: drop a stray seedPricing line left over from the 79f862b conflict resolution (applyCustomPricing is the only startup hook on this path on upstream) No functional change to the OpenRouter fetcher, the 24h refresh loop, or the parser updates.
Restore synchronous startup fallback seeding while keeping the initial network refresh asynchronous, and remove the periodic custom-pricing map write that raced with request reads. Track OpenRouter aliases so refreshes can remove obsolete bare names locally and in PostgreSQL, preserve ordered LiteLLM precedence, and accept free-model zero prices. VALID (fixed): #1, #2, #3, kenn-io#4, kenn-io#5 INVALID (dismissed): none PEDANTIC (skipped): none
MergePricing treated a zero rate from the higher-priority source as missing and filled it from a later catalog. Zero is valid pricing for free models, and ModelPricing cannot distinguish an explicit zero from an absent field, so a lower-priority nonzero row could silently turn a free model into a paid one. The first source to declare a pattern now owns the row entirely; later sources only contribute new patterns. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ReconcileModelPricing and SetPricingMeta were separate writes, so a crash or a concurrent pg push between them could observe OpenRouter alias rows without the _openrouter_aliases provenance metadata, leaving obsolete aliases unretireable on later refreshes. The metadata sentinel now commits inside the reconciliation transaction (matching the PostgreSQL side, where the meta row already travels in the same reconcile transaction). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
OpenRouter bare aliases were generated from suffix uniqueness within the OpenRouter catalog alone, but an unqualified pricing key outranks a provider-qualified one during resolution. An alias like minimax-m3 therefore shadowed LiteLLM's minimax/MiniMax-M3 row for bare session model names, inverting the documented LiteLLM-first precedence. Alias rows are now dropped at refresh time when a higher-priority source already covers the same canonical model name (via a qualified row or an exact bare row); the stored alias provenance metadata reflects the suppressed set so stale aliases still retire cleanly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The pgtest-tagged integration test still called ReconcileModelPricing with the old two-argument signature, breaking compilation for the pgtest suite. Pass the alias metadata through the reconciliation call, which also exercises the new atomic metadata write. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Alias suppression only sees sources that succeeded in the current refresh, so a LiteLLM outage let OpenRouter aliases be written unsuppressed and shadow persisted LiteLLM rows for bare model names. When any higher-priority fetch fails, the refresh now drops all alias rows from the OpenRouter batch and leaves the stored alias set and its provenance metadata untouched: qualified rows still refresh, and alias additions or retirements wait for a fully successful refresh that can re-validate them. This also avoids alias flapping across transient outages. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
be88058 to
613dd38
Compare
roborev: Combined Review (
|
Alias suppression removed OpenRouter's bare alias rows when a higher-priority source already covered the model, but left the provider-qualified row in place. LiteLLM's minimax/MiniMax-M3 and OpenRouter's minimax/minimax-m3 canonicalize identically and rank equally, so a bare MiniMax-M3 lookup found two tied keys, was rejected as ambiguous, and priced the model at zero. SuppressShadowedOpenRouterRows replaces SuppressShadowedOpenRouterAliases and generalizes the rule: an unqualified OpenRouter row is shadowed by any earlier row with the same canonical name, and a qualified row is shadowed by an earlier row with the same canonical provider and name. Canonical collisions across different providers stay, since they are distinct vendor listings and dropping one would leave its own qualified id unpriced. The function now reports the dropped patterns, and the refresh adds them to the reconciliation removal list. A row stored by an earlier refresh, while the higher-priority source still lacked the model, is retired instead of lingering and keeping the lookup ambiguous.
Suppressed OpenRouter rows were deleted from SQLite but survived in PostgreSQL. pricingSyncChanges derives removals only from the _openrouter_aliases diff, and a shadowed qualified row such as minimax/minimax-m3 is not an alias, so pg push left it in place. pg serve then kept resolving the bare model name against two canonically identical rows, treating it as ambiguous and pricing it at zero. A second sentinel, _openrouter_shadowed, records the patterns each refresh suppressed. Unlike the alias list, which push targets diff against their own copy to find retired aliases, the shadowed list is absolute: every pattern on it must not exist, so PostgreSQL retires it without tracking history. Deletion still requires provenance rather than a set difference, because a pattern PostgreSQL holds and the local archive lacks is otherwise indistinguishable from one another machine pushed. ReconcileModelPricing now takes a variadic set of PricingMeta entries so both sentinels commit in the same transaction as the rows they describe.
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
|
So, this is not converging cleanly enough to keep patching finding-by-finding: new states are emerging faster than reviews/fixes are closing them. Assessment from Codex: I would keep the fetch/parser work but reset the reconciliation layer:
|
3 participants
Adds OpenRouter’s public model catalog as a secondary pricing source while keeping LiteLLM first for overlapping rates. Fresh servers seed the embedded fallback synchronously, then refresh both catalogs in the background at startup and every 24 hours. Custom model overrides remain authoritative without periodic concurrent reassignment.
OpenRouter text-output models, including free models, are converted to per-million-token rates. Unique bare aliases are tracked and reconciled when they become ambiguous, with the same deletion behavior propagated by PostgreSQL push sync.
Updates pricing and privacy documentation to describe source priority, fallback behavior, and outbound catalog requests.