feat: Add capability based routing - #106
Merged
Merged
Conversation
Pull the bandit adjudication pipeline out of routing/index.js into a standalone module: the 12-dim context-vector builder, tier-config candidate eligibility, the bandit pick, and the WS4.2 propensity collapse rule. Behavior-preserving — all existing routing/bandit/ propensity tests pass unchanged. Motivation: the off-policy evaluator (WS4 follow-up) must replay the live policy's exact context features and candidate rules against logged rows; that requires them as importable functions, not inline code. Contract tests pin the vector layout and TASK_TYPES order so drift breaks loudly instead of silently corrupting counterfactual estimates. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…verable The hash-embedding fallback is non-semantic — while active, semantic cache and kNN matching are effectively disabled (384-dim vectors never match 768-dim real entries). Previously one transient provider failure latched the fallback permanently until process restart, logged only at debug level: an invisible, unrecoverable degradation. Now: degradation logs a warn on every state transition, retries the provider on a cooldown (LYNKR_EMBEDDINGS_RETRY_COOLDOWN_MS, default 60s) instead of latching, exposes state via getEmbeddingStatus() on /metrics/semantic-cache, and supports LYNKR_EMBEDDINGS_STRICT=true for deployments that prefer fail-loud over a silently fake cache. Default behavior stays fail-soft. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Circuit recovery previously depended on the next live user request hitting the half-open provider — a user paid the latency of testing a dead upstream, and an idle provider never recovered at all. A background prober now sweeps every registered provider whose breaker is OPEN/HALF_OPEN and runs a cheap synthetic request through the breaker itself, so cockatiel's own half-open machinery closes or re-opens the circuit without live traffic involved. Closed breakers are never probed — zero cost while everything is healthy. Built-in probes cover the local providers (ollama /api/tags, llamacpp and lmstudio /v1/models) where dead-upstream hangs historically lived; cloud providers can opt in via registerHealthProbe(). Status exposed on /metrics/circuit-breakers. Knobs: LYNKR_HEALTH_PROBE_ENABLED, LYNKR_HEALTH_PROBE_INTERVAL_MS, LYNKR_HEALTH_PROBE_TIMEOUT_MS. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Even with the evidence gate satisfied, a deterministic slice of sessions (LYNKR_DEESCALATION_HOLDOUT_PCT, default 10) is now served at the original tier instead of being demoted. Their telemetry rows carry method '+deescalation_holdout' vs the demoted cohort's '+deescalated', giving a continuously-running baseline that proves the demotion rule stays net-positive — instead of trusting a one-time calibration. The bucket is an FNV-1a hash of the session fingerprint, so a session lands on the same side of the holdout every turn — clean cohorts, no tier flip-flopping caused by the holdout itself. Sessions without an identity are never held out. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The escalation ladder reacted to score drift, context overflow, risk keywords, and vision needs — but nothing watched for the pinned model thrashing: re-issuing the same tool call with the same input, or repeating the same text, turn after turn. Every frame of such a loop is a mid-tool-exchange pin serve (unconditional — tool-call IDs aren't provider-portable), so a cheap-tier model could burn turns indefinitely. A narrow detector (3 identical consecutive tool calls by name+input, or 3 identical normalized text blocks, tail-window scan only) now trips the same safe intervention the embedded-text triggers already use: serve the pin this turn, DROP it, and let the next turn boundary re-route fresh. Knobs: LYNKR_STUCK_DETECTOR_ENABLED, LYNKR_STUCK_TOOL_REPEATS, LYNKR_STUCK_TEXT_REPEATS. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
LYNKR_EMBEDDINGS_PROVIDER=onnx runs the embedding model inside the Lynkr process via @huggingface/transformers (optionalDependency, ~q8 INT8) — no external Ollama/llama.cpp server on the semantic-cache/kNN hot path, no network hop, no contention with completion traffic. Measured warm embeds: 8-10ms in-process. Model is the ONNX build of the same model the Ollama path defaults to (Xenova/nomic-embed-text-v1, 768-dim), so existing kNN index entries and cached vectors stay valid — no migration. Weights download once into ~/.lynkr/models (LYNKR_ONNX_CACHE_DIR to override); nothing is bundled in the npm package. A failed load degrades through the shared loud/recoverable machinery and retries — never a silent latch. Knobs: LYNKR_EMBEDDINGS_PROVIDER, LYNKR_ONNX_EMBEDDING_MODEL, LYNKR_ONNX_CACHE_DIR, LYNKR_ONNX_DTYPE. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Consumes what WS4.2 has logged on every telemetry row since it shipped (propensity, candidates, quality_score) to score counterfactual routing policies from production logs alone — no live traffic, no A/B test. Answers "would a different policy have done better?" before deploying it. - routing/ope.js: IPS, self-normalized IPS, doubly-robust (Dudik/ Langford/Li 2011), and weighted-DR estimators, with a propensity floor, Kish effective-sample-size reporting, and four reference policies (first/last candidate, uniform, current-bandit-greedy). - bandit.estimateReward(): LinUCB's own per-arm ridge model reused as the DR regression term r-hat — no second model to train. - telemetry: new additive `context` column persists the bandit's 12-dim feature vector per row (the one WS4 field that was missing for DR); threaded from decision._banditContext at all five record sites. - scripts/ope-report.js: CLI report over the last N days. Tests prove estimator correctness on synthetic ground truth, including the doubly-robust property itself: corrupted propensities bias IPS but DR with a correct reward model still recovers the true policy value. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…I export Phase 3 of ROUTING-NOTES: three production-hardening gaps closed, all SQLite/zero-dependency by design (Lynkr stays npm-install-and-go; the store shapes remain adapter-friendly if a shared Redis ever lands). - Budget pre-flight: the hardcoded flat-$0.01 gate is now a real estimate — payload token count priced at the blended average of the configured tier models, expected output from max_tokens. Floors at the old $0.01 so unpriced configs gate exactly as before. - Token-aware (TPM) rate limiting: per-user tokens-per-minute window in the budget DB, off unless LYNKR_TPM_LIMIT is set. Estimate -> true-up pattern: pre-flight gates on actual window consumption + this request's estimate; actual usage recorded post-response. Soft admission control, not a hard ceiling — documented overshoot of ~one in-flight request, same trade every major gateway makes. - OTel GenAI observability without the @opentelemetry dependency tree: gen_ai.client.token.usage / gen_ai.client.operation.duration exported via a minimal OTLP/HTTP JSON push (OTEL_EXPORTER_OTLP_ENDPOINT, off by default) plus gen_ai_* semconv aliases on the existing Prometheus surface. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MCP broker: Lynkr was MCP-client-only — it could call configured MCP servers but offered nothing for OTHER clients to call into. New HTTP surface (GET /v1/mcp/servers, GET /v1/mcp/tools aggregated + namespaced, POST /v1/mcp/tools/call) turns one Lynkr install into a team's single MCP access point; downstream server credentials stay in Lynkr's config and never reach callers. Security: off by default, requires LYNKR_MCP_BROKER_ENABLED + LYNKR_MCP_BROKER_TOKEN (bearer, constant-time compare); enabled-without-token fails CLOSED (503). Per-server failures are reported in responses, never silently dropped. Optional LYNKR_MCP_BROKER_SERVERS allowlist. Tier pinning: documentation/tier-pinning.md formalizes the model-picker- as-tier-selector technique (model-slots.js + openai-model-slots.js) as one documented, generalizable feature — the two client shapes (gateway- advertised ids vs real-catalog id+parameter combos), the capture-first methodology, and the always-fall-through-on-unrecognized rule. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same fix as fix/lockfile-public-registry, applied to this branch's lockfile (which includes @huggingface/transformers): all resolved URLs now point at registry.npmjs.org, zero npm.devsnc.com references. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ase0 # Conflicts: # documentation/README.md # package-lock.json # package.json
npm audit (high) flagged two leaves under the optional ONNX-embedder dependency: sharp 0.34.5 (libvips CVE-2026-33327/33328/35590/35591, fixed in 0.35.x) and onnxruntime-node's adm-zip <0.6.0 (crafted-ZIP 4GB allocation). @huggingface/transformers 4.2.0 is the latest release and still pins both, so npm overrides force the patched versions directly: sharp ^0.35.4, adm-zip ^0.6.0. Exposure context: Lynkr's transformers use is text feature-extraction only — sharp (image pipeline) is never invoked, and adm-zip belongs to onnxruntime's install tooling. Verified post-override: audit clean, the ONNX embedder loads and embeds (768-dim), its test suite passes, and the lockfile stays fully public-registry. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
veerareddyvishal144
changed the title
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| "version": 1, | ||
| "heads": ["reasoning", "codegen", "debugging", "tool_use"], | ||
| "notes": "Capability profiles are decoupled from the predictor (see src/routing/capabilities.js). Editing this file re-routes traffic with zero retraining — the HyDRA shortfall-matching port. Tier profiles are seeded to mirror the legacy scalar bands (SIMPLE 0-19, MEDIUM 20-50, COMPLEX 51-75, REASONING 76-100). This file is read once at boot (same as config/model-tiers.json) — restart to pick up edits. No env vars: everything shortfall needs lives here.", | ||
| "enabled": true, |
There was a problem hiding this comment.
When a weighted-routing request produces a different shortfall selection, the shipped enabled value immediately replaces the configured provider, model, and tier, causing upgrades to change routing behavior without the documented operator opt-in.
Suggested change
| "enabled": true, | |
| "enabled": false, |
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Greptile Summary
The PR introduces capability-based model routing, capability seed generation, and Fireworks provider support.
Confidence Score: 4/5
The PR is not yet safe to merge because capability-based shortfall routing remains active by default and can override operators’ configured routing without opt-in.
The shipped capability configuration enables shortfall routing, and the weighted request path applies a disagreeing shortfall result by replacing the selected provider, model, and tier.
Files Needing Attention: config/model-capabilities.json, src/routing/index.js
Important Files Changed
Flowchart
%%{init: {'theme': 'neutral'}}%% flowchart TD A[Weighted request analysis] --> B[Build capability requirement vector] B --> C[Collect configured tier models] C --> D[Resolve family capability profiles] D --> E[Calculate weighted shortfall] E --> F[Select cheapest covering model] F --> G{Shortfall routing enabled?} G -- Yes --> H[Replace provider, model, and tier] G -- No --> I[Keep legacy tier selection] H --> J[Apply downstream routing guards] I --> JReviews (2): Last reviewed commit: "Added Readme" | Re-trigger Greptile