diff --git a/documentation/README.md b/documentation/README.md index 2f3025e..bc18db1 100644 --- a/documentation/README.md +++ b/documentation/README.md @@ -20,6 +20,7 @@ Connect Lynkr to your development tools: - **[Claude Code CLI Setup](claude-code-cli.md)** - Configure Claude Code CLI to use Lynkr - **[Claude Desktop Setup](claude-desktop.md)** - Route the Claude Desktop app through Lynkr (macOS gateway profile, model picker as tier selector) +- **[Tier Pinning](tier-pinning.md)** - Use any desktop client's model picker as a routing tier selector (the generalized technique behind the Claude Desktop and Codex integrations) - **[opencode Integration](opencode.md)** - Per-tier context windows via `lynkr run opencode` (picker doubles as tier pin; compaction always matches the serving model) - **[Context Window Header](context-window-header.md)** - `X-Lynkr-Context-Window`: the served model's real context window on every response, so clients can compact against the model actually serving them - **[Codex CLI Setup](codex-cli.md)** - Configure OpenAI Codex CLI with Lynkr (config.toml, wire_api, troubleshooting) diff --git a/documentation/tier-pinning.md b/documentation/tier-pinning.md new file mode 100644 index 0000000..5df37f0 --- /dev/null +++ b/documentation/tier-pinning.md @@ -0,0 +1,85 @@ +# Tier Pinning via the Client's Model Picker + +Lynkr turns the model dropdown that desktop AI clients already have into a +**routing tier selector** — no client plugin, no custom UI, no config file on +the client machine. Pick a "model" in the picker; Lynkr reads that pick as a +tier pin and routes accordingly. To our knowledge this technique has no prior +art in other gateways (checked against LiteLLM, OpenRouter, Portkey, Kong, +and workweave/router as of 2026-09). + +## The idea + +Desktop clients (Claude Desktop, Codex/ChatGPT Desktop) send whatever model +id the user picked in their dropdown. Lynkr sits in front of the provider, so +it sees that id before any routing happens. Instead of treating the picker as +"which upstream model to call," Lynkr maps picker entries onto its four +routing tiers: + +| Picker intent | Tier | What Lynkr does | +|---|---|---| +| "Auto" entry | none | Full content-based tier routing (default behavior) | +| Top model | `REASONING` | Pin to the REASONING tier's configured model | +| Mid model | `COMPLEX` / `MEDIUM` | Pin to that tier's configured model | +| Small model | `SIMPLE` | Pin to the SIMPLE tier's configured model | + +A pin **bypasses content scoring** for that request: the user explicitly +asked for a capability class, and an explicit user choice beats a heuristic. +Unrecognized ids/values always resolve to *no pin* (fall through to normal +routing) — the resolver never guesses (see `documentation/claude-desktop.md` +for the incident-shaped reasoning behind exact-match-only resolution). + +## Two client shapes, two mechanisms + +### 1. Gateway-advertised ids (Claude Desktop) — `src/routing/model-slots.js` + +Claude Desktop populates its picker from the gateway's own `/v1/models` +response, but **validates ids against a fixed known set** — arbitrary ids +break the picker. So Lynkr advertises five real Claude model ids and maps +them to tiers (`claude-opus-5` → REASONING, `claude-sonnet-5` → COMPLEX, +`claude-sonnet-4-6` → MEDIUM, `claude-haiku-4-5-20251001` → SIMPLE, with +`claude-fable-5` as the "Auto" no-pin entry). Wired via +`src/api/claude-desktop-gateway.js` (advertises the list) and +`src/api/router.js` (resolves the pick back to a tier). + +### 2. Real-catalog ids + parameters (Codex/ChatGPT Desktop) — +`src/routing/openai-model-slots.js` + +Codex has no gateway hook for its picker — it always sends a **real** OpenAI +model id plus a `reasoning.effort` field (`minimal | low | medium | high`, +shown in the UI as effort labels like "Light"). Lynkr pins on the +`(model, effort)` **combination** instead of inventing ids. Confirmed live +(2026-08-28) against a captured Codex Desktop request. Wired via +`src/api/openai-router.js` on both `/v1/chat/completions` and +`/v1/responses`. + +## Generalizing to a new client + +The pattern reduces to three questions: + +1. **What does the client actually send** when the user changes the picker? + Capture one real request (a temp diagnostic log; remove it after). Never + guess from UI labels — the label "Light" turned out to be wire value + `"low"`, and an early guess about a model variant's meaning had to be + removed for lack of evidence. +2. **Can Lynkr control the picker's contents?** If yes (Claude Desktop + shape): advertise ids the client will accept, map id → tier. If no + (Codex shape): map the real `(model, parameters)` combinations the picker + can produce. +3. **What happens on unrecognized input?** Always: no pin, fall through to + content scoring. A picker pin is advisory-input-turned-explicit — an + unmapped value must degrade to the default, never to a wrong pin. + +Both resolvers return `null` for anything unrecognized, and both feed the +same downstream pin mechanism (`_forceProvider`), so a new client mapping is +one small module + one resolver call at its ingress point. + +## Scope and caveats + +- A picker pin governs the **first model call of a turn**. Multi-step + agentic turns (tool loops) re-score by content on subsequent steps — a + property of the shared pin plumbing (`_forceProvider` is consumed on first + read), consistent across both client shapes. +- Pins select a **tier**, whose model comes from your `TIER_*` config — the + picker never routes to a model you haven't configured. +- Session pins (sticky routing) and picker pins compose: the picker pin wins + for the request that carries it, and normal stickiness resumes after. diff --git a/package-lock.json b/package-lock.json index 36fecc5..e0c5a87 100644 --- a/package-lock.json +++ b/package-lock.json @@ -38,6 +38,7 @@ "node": ">=20.0.0" }, "optionalDependencies": { + "@huggingface/transformers": "^4.2.0", "better-sqlite3": "^12.11.1", "dockerode": "^5.0.1", "tree-sitter": "^0.21.1", @@ -211,6 +212,37 @@ "node": ">=6" } }, + "node_modules/@huggingface/jinja": { + "version": "0.5.9", + "resolved": "https://registry.npmjs.org/@huggingface/jinja/-/jinja-0.5.9.tgz", + "integrity": "sha512-uWTG+l3VJRsl7EXxYizuL3P+cCPoc3cRqbWWRcQN0FhejRfbdq0RNhCmbY/YDtnTcz9icdLYuLDjsnz4d8JMuw==", + "license": "MIT", + "optional": true, + "engines": { + "node": ">=18" + } + }, + "node_modules/@huggingface/tokenizers": { + "version": "0.1.3", + "resolved": "https://registry.npmjs.org/@huggingface/tokenizers/-/tokenizers-0.1.3.tgz", + "integrity": "sha512-8rF/RRT10u+kn7YuUbUg0OF30K8rjTc78aHpxT+qJ1uWSqxT1MHi8+9ltwYfkFYJzT/oS+qw3JVfHtNMGAdqyA==", + "license": "Apache-2.0", + "optional": true + }, + "node_modules/@huggingface/transformers": { + "version": "4.2.0", + "resolved": "https://registry.npmjs.org/@huggingface/transformers/-/transformers-4.2.0.tgz", + "integrity": "sha512-8BRCoBMH0XsWaEIamuR0LrJGAfftgHAfb2Vrffy0VKlSAE/MnUJ5/h/zTfEP3fDIft+nk7TqB8xXEyABGitBjQ==", + "license": "Apache-2.0", + "optional": true, + "dependencies": { + "@huggingface/jinja": "^0.5.6", + "@huggingface/tokenizers": "^0.1.3", + "onnxruntime-node": "1.24.3", + "onnxruntime-web": "1.26.0-dev.20260416-b7804b056c", + "sharp": "^0.34.5" + } + }, "node_modules/@humanwhocodes/config-array": { "version": "0.13.0", "resolved": "https://registry.npmjs.org/@humanwhocodes/config-array/-/config-array-0.13.0.tgz", @@ -219,35 +251,594 @@ "dev": true, "license": "Apache-2.0", "dependencies": { - "@humanwhocodes/object-schema": "^2.0.3", - "debug": "^4.3.1", - "minimatch": "^3.0.5" + "@humanwhocodes/object-schema": "^2.0.3", + "debug": "^4.3.1", + "minimatch": "^3.0.5" + }, + "engines": { + "node": ">=10.10.0" + } + }, + "node_modules/@humanwhocodes/module-importer": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/@humanwhocodes/module-importer/-/module-importer-1.0.1.tgz", + "integrity": "sha512-bxveV4V8v5Yb4ncFTT3rPSgZBOpCkjfK0y4oVVVJwIuDVBRMDXrPyXRL988i5ap9m9bnyEEjWfm5WkBmtffLfA==", + "dev": true, + "license": "Apache-2.0", + "engines": { + "node": ">=12.22" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/nzakas" + } + }, + "node_modules/@humanwhocodes/object-schema": { + "version": "2.0.3", + "resolved": "https://registry.npmjs.org/@humanwhocodes/object-schema/-/object-schema-2.0.3.tgz", + "integrity": "sha512-93zYdMES/c1D69yZiKDBj0V24vqNzB/koF26KPaagAfd3P/4gUlh3Dys5ogAK+Exi9QyzlD8x/08Zt7wIKcDcA==", + "deprecated": "Use @eslint/object-schema instead", + "dev": true, + "license": "BSD-3-Clause" + }, + "node_modules/@img/colour": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/@img/colour/-/colour-1.1.0.tgz", + "integrity": "sha512-Td76q7j57o/tLVdgS746cYARfSyxk8iEfRxewL9h4OMzYhbW4TAcppl0mT4eyqXddh6L/jwoM75mo7ixa/pCeQ==", + "license": "MIT", + "optional": true, + "engines": { + "node": ">=18" + } + }, + "node_modules/@img/sharp-darwin-arm64": { + "version": "0.35.4", + "resolved": "https://registry.npmjs.org/@img/sharp-darwin-arm64/-/sharp-darwin-arm64-0.35.4.tgz", + "integrity": "sha512-Uhfl4V4lhP2nbUVF9+hyH1+luj86f1gUFeo8ALYxFoULoU+G87D43BfeMP8XHsk9boxAnCY/bf2EHwhA7MuGsA==", + "cpu": [ + "arm64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-darwin-arm64": "1.3.3" + } + }, + "node_modules/@img/sharp-darwin-x64": { + "version": "0.35.4", + "resolved": "https://registry.npmjs.org/@img/sharp-darwin-x64/-/sharp-darwin-x64-0.35.4.tgz", + "integrity": "sha512-hWniXY3bG5qKpkKrAwPe4y+VTPmf086YQAnkxWh7uA1YrlRouWGa0M0Mxj3ZjnXFkv7/TD1bTy9lGUK26vRvWw==", + "cpu": [ + "x64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-darwin-x64": "1.3.3" + } + }, + "node_modules/@img/sharp-freebsd-wasm32": { + "version": "0.35.4", + "resolved": "https://registry.npmjs.org/@img/sharp-freebsd-wasm32/-/sharp-freebsd-wasm32-0.35.4.tgz", + "integrity": "sha512-lIsKw/BU+kjB4eZjxrYrZmwOJYi3Ajrv66iAlBmUPyKc3HpnloevB1g3wxGD9P/5BbQ1brBGl65VRRrCvQDEqA==", + "license": "Apache-2.0", + "optional": true, + "os": [ + "freebsd" + ], + "dependencies": { + "@img/sharp-wasm32": "0.35.4" + }, + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-darwin-arm64": { + "version": "1.3.3", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-darwin-arm64/-/sharp-libvips-darwin-arm64-1.3.3.tgz", + "integrity": "sha512-suTBPTDGrI9WodccaDdwZItTSaBYASlBk1NSfElSHrUfzu3szG6lvIF58+WiFvnfzuK8ZBFS5zE00PxqxnRiPg==", + "cpu": [ + "arm64" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "darwin" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-darwin-x64": { + "version": "1.3.3", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-darwin-x64/-/sharp-libvips-darwin-x64-1.3.3.tgz", + "integrity": "sha512-FVJZ5mITMobmXIz/hPDTw0EintTW5H3WfrxwLqEqjiIihlu+hVRyGrFQ60xl0Lxn7Bt3zdpevPaQi0HEzqz9fw==", + "cpu": [ + "x64" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "darwin" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-linux-arm": { + "version": "1.3.3", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-arm/-/sharp-libvips-linux-arm-1.3.3.tgz", + "integrity": "sha512-3rbU4vqXXc3hY/OiXdl52xZvT0F1yEngWfvqudtPJg/KkyiaQw2DRsFrNzpmLvfavbwOq3qXn36GP8obHRULQA==", + "cpu": [ + "arm" + ], + "libc": [ + "glibc" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-linux-arm64": { + "version": "1.3.3", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-arm64/-/sharp-libvips-linux-arm64-1.3.3.tgz", + "integrity": "sha512-0DaL0A6Xu6sQSQFwe4iVCrKWU2cCTItnRsYsCdxAMm9NF6twAA9BKnoqy4hqz4+azQ0JHuA26qiUKsf1XJ/v5A==", + "cpu": [ + "arm64" + ], + "libc": [ + "glibc" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-linux-ppc64": { + "version": "1.3.3", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-ppc64/-/sharp-libvips-linux-ppc64-1.3.3.tgz", + "integrity": "sha512-cdn1OvUBwsXhbC0zSzJnNzf5MZ/mTrobawDvNXBTxe8VtqKAm0sRuEY2Evzovb/w9JMk4TvRxqt1mekSuJz64w==", + "cpu": [ + "ppc64" + ], + "libc": [ + "glibc" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-linux-riscv64": { + "version": "1.3.3", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-riscv64/-/sharp-libvips-linux-riscv64-1.3.3.tgz", + "integrity": "sha512-HjPVx7yKz+0lqdhDlTw1tt90wamBoxhiXpvl1XZpJLiHH4RCJ5yDTqH+VlYPv2fwFs89JFw4c1IexYOcQUi4IQ==", + "cpu": [ + "riscv64" + ], + "libc": [ + "glibc" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-linux-s390x": { + "version": "1.3.3", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-s390x/-/sharp-libvips-linux-s390x-1.3.3.tgz", + "integrity": "sha512-neWLh+3yCNThxnfy3c4BbVBeGgt9aftno+XbT56iK28RgeDs3UOFWviLWlUu0bArYVYJaFDK+RRohbicUNCm8Q==", + "cpu": [ + "s390x" + ], + "libc": [ + "glibc" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-linux-x64": { + "version": "1.3.3", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-x64/-/sharp-libvips-linux-x64-1.3.3.tgz", + "integrity": "sha512-4vKmvAst9nrowcqquKFAyZJUDolUaIp8uRiN0mWFguJ1IplC9/pitXtlnnlU4aa/eJw3J7i67V+pwUL+wZGdsA==", + "cpu": [ + "x64" + ], + "libc": [ + "glibc" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-linuxmusl-arm64": { + "version": "1.3.3", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linuxmusl-arm64/-/sharp-libvips-linuxmusl-arm64-1.3.3.tgz", + "integrity": "sha512-Y9kQaLMuNoB0bPYOOdcZMaseNrFpPodIWWMrx+CZyydf2xn68j9WYc6sWWRrDwNkzCQjKYfc68L7jKjGlHMibw==", + "cpu": [ + "arm64" + ], + "libc": [ + "musl" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-linuxmusl-x64": { + "version": "1.3.3", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linuxmusl-x64/-/sharp-libvips-linuxmusl-x64-1.3.3.tgz", + "integrity": "sha512-fj8Mv0HHfD1Rr+4I68+3agJynxDWtBFgicTbSOb9Bke6pIwzGcJ+RX/yHjmiEGFMCavY/dxvem7MyNaJF+wDiw==", + "cpu": [ + "x64" + ], + "libc": [ + "musl" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-linux-arm": { + "version": "0.35.4", + "resolved": "https://registry.npmjs.org/@img/sharp-linux-arm/-/sharp-linux-arm-0.35.4.tgz", + "integrity": "sha512-7OAS8gI0EReKGVN2HssHlM6umJgxF5VI3xN0p9FA91p/YO+ou5hiNghLdZ5BEHztwaaK5+bLKRf8x/o2L2nk9A==", + "cpu": [ + "arm" + ], + "libc": [ + "glibc" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linux-arm": "1.3.3" + } + }, + "node_modules/@img/sharp-linux-arm64": { + "version": "0.35.4", + "resolved": "https://registry.npmjs.org/@img/sharp-linux-arm64/-/sharp-linux-arm64-0.35.4.tgz", + "integrity": "sha512-De4jpEnAU8Hd5oT0j1G3uL4ZvTuipVMn7YC6vPaJhy6/7EwEae0SVAoBrUMYQbkLGDm85taVWwuPc1a44LTzCQ==", + "cpu": [ + "arm64" + ], + "libc": [ + "glibc" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linux-arm64": "1.3.3" + } + }, + "node_modules/@img/sharp-linux-ppc64": { + "version": "0.35.4", + "resolved": "https://registry.npmjs.org/@img/sharp-linux-ppc64/-/sharp-linux-ppc64-0.35.4.tgz", + "integrity": "sha512-2oYZJeIl4kCcMGk4ouZVjnkCtFrpQFlNEtJ6GbxzhHQchwH0NH/qEb9ykmOl29dqwMq+JhFdZn+1ak2FKhI9fQ==", + "cpu": [ + "ppc64" + ], + "libc": [ + "glibc" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linux-ppc64": "1.3.3" + } + }, + "node_modules/@img/sharp-linux-riscv64": { + "version": "0.35.4", + "resolved": "https://registry.npmjs.org/@img/sharp-linux-riscv64/-/sharp-linux-riscv64-0.35.4.tgz", + "integrity": "sha512-cPbNChoRURAWdebDIHSenxRpgEdy7JkPydSnUxRm9VvKD7m0/xVaR/8Fzlu81pk5nHEvHH87UZUA7cTtwnbJSA==", + "cpu": [ + "riscv64" + ], + "libc": [ + "glibc" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linux-riscv64": "1.3.3" + } + }, + "node_modules/@img/sharp-linux-s390x": { + "version": "0.35.4", + "resolved": "https://registry.npmjs.org/@img/sharp-linux-s390x/-/sharp-linux-s390x-0.35.4.tgz", + "integrity": "sha512-RY0JFY8Fd6RonCBtHz+DvadaPkXDSI1AUn6yWL9TipqkZ1vY8w8evqdgyDFnkm4/K1ve1TvZiaePP5oSd4+WVQ==", + "cpu": [ + "s390x" + ], + "libc": [ + "glibc" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linux-s390x": "1.3.3" + } + }, + "node_modules/@img/sharp-linux-x64": { + "version": "0.35.4", + "resolved": "https://registry.npmjs.org/@img/sharp-linux-x64/-/sharp-linux-x64-0.35.4.tgz", + "integrity": "sha512-9qvvEAuk8k89TfWUoX2htWjbAMX8p+NxCppjpcg5k6xMsjhBQPTsoIh36h9Qde4WRuGpJeYnOjdosDn/cnv+OA==", + "cpu": [ + "x64" + ], + "libc": [ + "glibc" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linux-x64": "1.3.3" + } + }, + "node_modules/@img/sharp-linuxmusl-arm64": { + "version": "0.35.4", + "resolved": "https://registry.npmjs.org/@img/sharp-linuxmusl-arm64/-/sharp-linuxmusl-arm64-0.35.4.tgz", + "integrity": "sha512-KB5jxpfWQTr0nc3xdHtWChdbifHrBGsd2SM62Eyxrl8afikm+f5qGBU75SJIZBT/S1MC8XyacdlXBMSWq6OURA==", + "cpu": [ + "arm64" + ], + "libc": [ + "musl" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linuxmusl-arm64": "1.3.3" + } + }, + "node_modules/@img/sharp-linuxmusl-x64": { + "version": "0.35.4", + "resolved": "https://registry.npmjs.org/@img/sharp-linuxmusl-x64/-/sharp-linuxmusl-x64-0.35.4.tgz", + "integrity": "sha512-f+eZJZIQNEEd26RPSW+76chwOf1XtA2Y/O+5ocVyLliHkeih3e+jhLVBdNTd2rS3IbNXK8+ug93Vf5ZXtF5Lxg==", + "cpu": [ + "x64" + ], + "libc": [ + "musl" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linuxmusl-x64": "1.3.3" + } + }, + "node_modules/@img/sharp-wasm32": { + "version": "0.35.4", + "resolved": "https://registry.npmjs.org/@img/sharp-wasm32/-/sharp-wasm32-0.35.4.tgz", + "integrity": "sha512-zQnl4Kwp7Q6NHsENtU2T/00Zi+w3AQNwz3+UaTyVBy2FpXrzXzGjndpK61onhZjRtRpQXxCTeqw19bVyXOh7jA==", + "license": "Apache-2.0 AND LGPL-3.0-or-later AND MIT", + "optional": true, + "dependencies": { + "@emnapi/runtime": "^1.11.3" }, "engines": { - "node": ">=10.10.0" + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" } }, - "node_modules/@humanwhocodes/module-importer": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/@humanwhocodes/module-importer/-/module-importer-1.0.1.tgz", - "integrity": "sha512-bxveV4V8v5Yb4ncFTT3rPSgZBOpCkjfK0y4oVVVJwIuDVBRMDXrPyXRL988i5ap9m9bnyEEjWfm5WkBmtffLfA==", - "dev": true, + "node_modules/@img/sharp-wasm32/node_modules/@emnapi/runtime": { + "version": "1.11.3", + "resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.11.3.tgz", + "integrity": "sha512-Xz4Tpyki7XyrpbUK1jR1AhdAdaXyhhY4lZ3neLodmhpuWfy2PAQN5B46sAiU4liOXGLkHypn/qU+jvfWSCYYLA==", + "license": "MIT", + "optional": true, + "dependencies": { + "tslib": "^2.4.0" + } + }, + "node_modules/@img/sharp-webcontainers-wasm32": { + "version": "0.35.4", + "resolved": "https://registry.npmjs.org/@img/sharp-webcontainers-wasm32/-/sharp-webcontainers-wasm32-0.35.4.tgz", + "integrity": "sha512-ESfNkywmCfPNyaZjxooddJQiQ+l/nTpGEOGthxiLnIHXC/CmcBixnfwUleX9mCz9ovrUUvKMap/pm8RYbzfwaA==", + "cpu": [ + "wasm32" + ], "license": "Apache-2.0", + "optional": true, + "dependencies": { + "@img/sharp-wasm32": "0.35.4" + }, "engines": { - "node": ">=12.22" + "node": ">=20.9.0" }, "funding": { - "type": "github", - "url": "https://github.com/sponsors/nzakas" + "url": "https://opencollective.com/libvips" } }, - "node_modules/@humanwhocodes/object-schema": { - "version": "2.0.3", - "resolved": "https://registry.npmjs.org/@humanwhocodes/object-schema/-/object-schema-2.0.3.tgz", - "integrity": "sha512-93zYdMES/c1D69yZiKDBj0V24vqNzB/koF26KPaagAfd3P/4gUlh3Dys5ogAK+Exi9QyzlD8x/08Zt7wIKcDcA==", - "deprecated": "Use @eslint/object-schema instead", - "dev": true, - "license": "BSD-3-Clause" + "node_modules/@img/sharp-win32-arm64": { + "version": "0.35.4", + "resolved": "https://registry.npmjs.org/@img/sharp-win32-arm64/-/sharp-win32-arm64-0.35.4.tgz", + "integrity": "sha512-iNdlBX9gLVvqe2I3uIJSIKTq6wckP/DYxZtcqxm09x5Gi24DnFBmPAWZmr60ZyYMG0xlzo6goG3670ar+RXvRw==", + "cpu": [ + "arm64" + ], + "license": "Apache-2.0 AND LGPL-3.0-or-later", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-win32-ia32": { + "version": "0.35.4", + "resolved": "https://registry.npmjs.org/@img/sharp-win32-ia32/-/sharp-win32-ia32-0.35.4.tgz", + "integrity": "sha512-kqRsbaa5CS6KHlpxnN7WhE6vAAugXyZButpRdvDWetlv6Qv4N9WTcrWzF7tXfB9T7MsoadqdI8hmwLq6UlLvtw==", + "cpu": [ + "ia32" + ], + "license": "Apache-2.0 AND LGPL-3.0-or-later", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": "^20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-win32-x64": { + "version": "0.35.4", + "resolved": "https://registry.npmjs.org/@img/sharp-win32-x64/-/sharp-win32-x64-0.35.4.tgz", + "integrity": "sha512-XtmnYhBcrORsJ4XJngyzr/EWP0hRZLAZRFaApdKuviyqF78+ylxh2y06ZmtULAMOnObJ3ucpN0AcwSWnMowTRg==", + "cpu": [ + "x64" + ], + "license": "Apache-2.0 AND LGPL-3.0-or-later", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + } }, "node_modules/@inquirer/ansi": { "version": "2.0.8", @@ -2081,6 +2672,16 @@ "acorn": "^6.0.0 || ^7.0.0 || ^8.0.0" } }, + "node_modules/adm-zip": { + "version": "0.6.0", + "resolved": "https://registry.npmjs.org/adm-zip/-/adm-zip-0.6.0.tgz", + "integrity": "sha512-XleryMhbuksdKtofnWZ9Sk+4CUTbms4Mb/EU32SZwToAyZ5RgVos/ki8n+yr0LWHOGKuakbXTuuYNHLQjhddgg==", + "license": "MIT", + "optional": true, + "engines": { + "node": ">=14.0" + } + }, "node_modules/ajv": { "version": "6.15.0", "resolved": "https://registry.npmjs.org/ajv/-/ajv-6.15.0.tgz", @@ -2294,6 +2895,14 @@ "url": "https://opencollective.com/express" } }, + "node_modules/boolean": { + "version": "3.2.0", + "resolved": "https://registry.npmjs.org/boolean/-/boolean-3.2.0.tgz", + "integrity": "sha512-d0II/GO9uf9lfUHH2BQsjxzRJZBdsjgsBiW4BvhWk/3qoKwQFjIDVN19PfX8F2D/r9PCMTtLWjYVCFrpeYUzsw==", + "deprecated": "Package no longer supported. Contact Support at https://www.npmjs.com/support for more info.", + "license": "MIT", + "optional": true + }, "node_modules/brace-expansion": { "version": "1.1.18", "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.18.tgz", @@ -2697,6 +3306,42 @@ "dev": true, "license": "MIT" }, + "node_modules/define-data-property": { + "version": "1.1.4", + "resolved": "https://registry.npmjs.org/define-data-property/-/define-data-property-1.1.4.tgz", + "integrity": "sha512-rBMvIzlpA8v6E+SJZoo++HAYqsLrkg7MSfIinMPFhmkorw7X+dOXVJQs+QT69zGkzMyfDnIMN2Wid1+NbL3T+A==", + "license": "MIT", + "optional": true, + "dependencies": { + "es-define-property": "^1.0.0", + "es-errors": "^1.3.0", + "gopd": "^1.0.1" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/define-properties": { + "version": "1.2.1", + "resolved": "https://registry.npmjs.org/define-properties/-/define-properties-1.2.1.tgz", + "integrity": "sha512-8QmQKqEASLd5nx0U1B1okLElbUuuttJ/AnYmRXbbbGDWh6uS208EjD4Xqq/I9wK7u0v6O08XhTWnt5XtEbR6Dg==", + "license": "MIT", + "optional": true, + "dependencies": { + "define-data-property": "^1.0.1", + "has-property-descriptors": "^1.0.0", + "object-keys": "^1.1.1" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, "node_modules/depd": { "version": "2.0.0", "resolved": "https://registry.npmjs.org/depd/-/depd-2.0.0.tgz", @@ -2716,6 +3361,13 @@ "node": ">=8" } }, + "node_modules/detect-node": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/detect-node/-/detect-node-2.1.0.tgz", + "integrity": "sha512-T0NIuQpnTvFDATNuHN5roPwSBG83rFsuO+MXXH9/3N1eFbn4wcPjttvjMLEPWJ0RGUYgQE7cGgS3tNxbqCGM7g==", + "license": "MIT", + "optional": true + }, "node_modules/diff": { "version": "5.2.2", "resolved": "https://registry.npmjs.org/diff/-/diff-5.2.2.tgz", @@ -2873,6 +3525,13 @@ "tests/browser-compat" ] }, + "node_modules/es6-error": { + "version": "4.1.1", + "resolved": "https://registry.npmjs.org/es6-error/-/es6-error-4.1.1.tgz", + "integrity": "sha512-Um/+FxMr9CISWh0bi5Zv0iOD+4cFh5qLeks1qhAopKVAJw3drgKbKySikp7wGhDL0HPeaja0P5ULZrxLkniUVg==", + "license": "MIT", + "optional": true + }, "node_modules/escalade": { "version": "3.2.0", "resolved": "https://registry.npmjs.org/escalade/-/escalade-3.2.0.tgz", @@ -2893,7 +3552,7 @@ "version": "4.0.0", "resolved": "https://registry.npmjs.org/escape-string-regexp/-/escape-string-regexp-4.0.0.tgz", "integrity": "sha512-TtpcNJ3XAzx3Gq8sWRzJaVajRs0uVxA2YAkdb1jm2YkPz4G6egUFAyA3n5vtEIZefPk5Wa4UXbKuS5fKkJWdgA==", - "dev": true, + "devOptional": true, "license": "MIT", "engines": { "node": ">=10" @@ -3327,6 +3986,13 @@ "node": "^10.12.0 || >=12.0.0" } }, + "node_modules/flatbuffers": { + "version": "25.9.23", + "resolved": "https://registry.npmjs.org/flatbuffers/-/flatbuffers-25.9.23.tgz", + "integrity": "sha512-MI1qs7Lo4Syw0EOzUl0xjs2lsoeqFku44KpngfIduHBYvzm8h2+7K8YMQh1JtVVVrUvhLpNwqVi4DERegUJhPQ==", + "license": "Apache-2.0", + "optional": true + }, "node_modules/flatted": { "version": "3.4.4", "resolved": "https://registry.npmjs.org/flatted/-/flatted-3.4.4.tgz", @@ -3478,6 +4144,24 @@ "node": ">=10.13.0" } }, + "node_modules/global-agent": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/global-agent/-/global-agent-3.0.0.tgz", + "integrity": "sha512-PT6XReJ+D07JvGoxQMkT6qji/jVNfX/h364XHZOWeRzy64sSFr+xJ5OX7LI3b4MPQzdL4H8Y8M0xzPpsVMwA8Q==", + "license": "BSD-3-Clause", + "optional": true, + "dependencies": { + "boolean": "^3.0.1", + "es6-error": "^4.1.1", + "matcher": "^3.0.0", + "roarr": "^2.15.3", + "semver": "^7.3.2", + "serialize-error": "^7.0.1" + }, + "engines": { + "node": ">=10.0" + } + }, "node_modules/globals": { "version": "13.24.0", "resolved": "https://registry.npmjs.org/globals/-/globals-13.24.0.tgz", @@ -3494,6 +4178,23 @@ "url": "https://github.com/sponsors/sindresorhus" } }, + "node_modules/globalthis": { + "version": "1.0.4", + "resolved": "https://registry.npmjs.org/globalthis/-/globalthis-1.0.4.tgz", + "integrity": "sha512-DpLKbNU4WylpxJykQujfCcwYWiV/Jhm50Goo0wrVILAv5jOr9d+H+UR3PhSCD2rCCEIg0uc+G+muBTwD54JhDQ==", + "license": "MIT", + "optional": true, + "dependencies": { + "define-properties": "^1.2.1", + "gopd": "^1.0.1" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, "node_modules/gopd": { "version": "1.2.0", "resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz", @@ -3513,6 +4214,13 @@ "dev": true, "license": "MIT" }, + "node_modules/guid-typescript": { + "version": "1.0.9", + "resolved": "https://registry.npmjs.org/guid-typescript/-/guid-typescript-1.0.9.tgz", + "integrity": "sha512-Y8T4vYhEfwJOTbouREvG+3XDsjr8E3kIr7uf+JZ0BYloFsttiHU0WfvANVsR7TxNUJa/WpCnw/Ino/p+DeBhBQ==", + "license": "ISC", + "optional": true + }, "node_modules/has-flag": { "version": "4.0.0", "resolved": "https://registry.npmjs.org/has-flag/-/has-flag-4.0.0.tgz", @@ -3523,6 +4231,19 @@ "node": ">=8" } }, + "node_modules/has-property-descriptors": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/has-property-descriptors/-/has-property-descriptors-1.0.2.tgz", + "integrity": "sha512-55JNKuIW+vq4Ke1BjOTjM2YctQIvCT7GFzHwmfZPGo5wnrgkid0YQtnAleFSqumZm4az3n2BS+erby5ipJdgrg==", + "license": "MIT", + "optional": true, + "dependencies": { + "es-define-property": "^1.0.0" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, "node_modules/has-symbols": { "version": "1.1.0", "resolved": "https://registry.npmjs.org/has-symbols/-/has-symbols-1.1.0.tgz", @@ -3850,6 +4571,13 @@ "dev": true, "license": "MIT" }, + "node_modules/json-stringify-safe": { + "version": "5.0.1", + "resolved": "https://registry.npmjs.org/json-stringify-safe/-/json-stringify-safe-5.0.1.tgz", + "integrity": "sha512-ZClg6AaYvamvYEE82d3Iyd3vSSIjQ+odgjaTzRuO3s7toCdFKczob2i0zCh7JE8kWn17yvAWhUVxvqGwUalsRA==", + "license": "ISC", + "optional": true + }, "node_modules/json-with-bigint": { "version": "3.5.12", "resolved": "https://registry.npmjs.org/json-with-bigint/-/json-with-bigint-3.5.12.tgz", @@ -3918,6 +4646,19 @@ "license": "Apache-2.0", "optional": true }, + "node_modules/matcher": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/matcher/-/matcher-3.0.0.tgz", + "integrity": "sha512-OkeDaAZ/bQCxeFAozM55PKcKU0yJMPGifLwV4Qgjitu+5MoAfSQN4lsLJeXZ1b8w0x+/Emda6MZgXS1jvsapng==", + "license": "MIT", + "optional": true, + "dependencies": { + "escape-string-regexp": "^4.0.0" + }, + "engines": { + "node": ">=10" + } + }, "node_modules/math-intrinsics": { "version": "1.1.0", "resolved": "https://registry.npmjs.org/math-intrinsics/-/math-intrinsics-1.1.0.tgz", @@ -4233,6 +4974,16 @@ "url": "https://github.com/sponsors/ljharb" } }, + "node_modules/object-keys": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/object-keys/-/object-keys-1.1.1.tgz", + "integrity": "sha512-NuAESUOUMrlIXOfHKzD6bpPu3tYt3xvjNdRIQ+FeT0lNb4K8WR70CaDxhuNguS2XG+GjkyMwOzsN5ZktImfhLA==", + "license": "MIT", + "optional": true, + "engines": { + "node": ">= 0.4" + } + }, "node_modules/obug": { "version": "2.1.4", "resolved": "https://registry.npmjs.org/obug/-/obug-2.1.4.tgz", @@ -4277,6 +5028,53 @@ "wrappy": "1" } }, + "node_modules/onnxruntime-common": { + "version": "1.24.3", + "resolved": "https://registry.npmjs.org/onnxruntime-common/-/onnxruntime-common-1.24.3.tgz", + "integrity": "sha512-GeuPZO6U/LBJXvwdaqHbuUmoXiEdeCjWi/EG7Y1HNnDwJYuk6WUbNXpF6luSUY8yASul3cmUlLGrCCL1ZgVXqA==", + "license": "MIT", + "optional": true + }, + "node_modules/onnxruntime-node": { + "version": "1.24.3", + "resolved": "https://registry.npmjs.org/onnxruntime-node/-/onnxruntime-node-1.24.3.tgz", + "integrity": "sha512-JH7+czbc8ALA819vlTgcV+Q214/+VjGeBHDjX81+ZCD0PCVCIFGFNtT0V4sXG/1JXypKPgScQcB3ij/hk3YnTg==", + "hasInstallScript": true, + "license": "MIT", + "optional": true, + "os": [ + "win32", + "darwin", + "linux" + ], + "dependencies": { + "adm-zip": "^0.5.16", + "global-agent": "^3.0.0", + "onnxruntime-common": "1.24.3" + } + }, + "node_modules/onnxruntime-web": { + "version": "1.26.0-dev.20260416-b7804b056c", + "resolved": "https://registry.npmjs.org/onnxruntime-web/-/onnxruntime-web-1.26.0-dev.20260416-b7804b056c.tgz", + "integrity": "sha512-MD6Ss4GSpQBo6zqoJzyT9LRbKYs7x/JVN23FT24EcEvlqF4VuzPOeH6X38orZPKHQDbprn7K+SBpu0/mj2CQiw==", + "license": "MIT", + "optional": true, + "dependencies": { + "flatbuffers": "^25.1.24", + "guid-typescript": "^1.0.9", + "long": "^5.2.3", + "onnxruntime-common": "1.24.0-dev.20251116-b39e144322", + "platform": "^1.3.6", + "protobufjs": "^7.2.4" + } + }, + "node_modules/onnxruntime-web/node_modules/onnxruntime-common": { + "version": "1.24.0-dev.20251116-b39e144322", + "resolved": "https://registry.npmjs.org/onnxruntime-common/-/onnxruntime-common-1.24.0-dev.20251116-b39e144322.tgz", + "integrity": "sha512-BOoomdHYmNRL5r4iQ4bMvsl2t0/hzVQ3OM3PHD0gxeXu1PmggqBv3puZicEUVOA3AtHHYmqZtjMj9FOfGrATTw==", + "license": "MIT", + "optional": true + }, "node_modules/optionator": { "version": "0.9.4", "resolved": "https://registry.npmjs.org/optionator/-/optionator-0.9.4.tgz", @@ -4579,6 +5377,13 @@ "integrity": "sha512-cHjPPsE+vhj/tnhCy/wiMh3M3z3h/j15zHQX+S9GkTBgqJuTuJzYJ4gUyACLhDaJ7kk9ba9iRDmbH2tJU03OiA==", "license": "MIT" }, + "node_modules/platform": { + "version": "1.3.6", + "resolved": "https://registry.npmjs.org/platform/-/platform-1.3.6.tgz", + "integrity": "sha512-fnWVljUchTro6RiCFvCXBbNhJc2NijN7oIQxbwsyL0buWJPG85v81ehlHI9fXrJsMNgTofEoWIQeClKpgxFLrg==", + "license": "MIT", + "optional": true + }, "node_modules/prebuild-install": { "version": "7.1.3", "resolved": "https://registry.npmjs.org/prebuild-install/-/prebuild-install-7.1.3.tgz", @@ -4879,6 +5684,24 @@ "url": "https://github.com/sponsors/isaacs" } }, + "node_modules/roarr": { + "version": "2.15.4", + "resolved": "https://registry.npmjs.org/roarr/-/roarr-2.15.4.tgz", + "integrity": "sha512-CHhPh+UNHD2GTXNYhPWLnU8ONHdI+5DI+4EYIAOaiD63rHeYlZvyh8P+in5999TTSFgUYuKUAjzRI4mdh/p+2A==", + "license": "BSD-3-Clause", + "optional": true, + "dependencies": { + "boolean": "^3.0.1", + "detect-node": "^2.0.4", + "globalthis": "^1.0.1", + "json-stringify-safe": "^5.0.1", + "semver-compare": "^1.0.0", + "sprintf-js": "^1.1.2" + }, + "engines": { + "node": ">=8.0" + } + }, "node_modules/router": { "version": "2.2.0", "resolved": "https://registry.npmjs.org/router/-/router-2.2.0.tgz", @@ -4974,6 +5797,13 @@ "node": ">=10" } }, + "node_modules/semver-compare": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/semver-compare/-/semver-compare-1.0.0.tgz", + "integrity": "sha512-YM3/ITh2MJ5MtzaM429anh+x2jiLVjqILF4m4oyQB18W7Ggea7BfqdH/wGMK7dDiMghv/6WG7znWMwUDzJiXow==", + "license": "MIT", + "optional": true + }, "node_modules/send": { "version": "1.2.1", "resolved": "https://registry.npmjs.org/send/-/send-1.2.1.tgz", @@ -5000,6 +5830,35 @@ "url": "https://opencollective.com/express" } }, + "node_modules/serialize-error": { + "version": "7.0.1", + "resolved": "https://registry.npmjs.org/serialize-error/-/serialize-error-7.0.1.tgz", + "integrity": "sha512-8I8TjW5KMOKsZQTvoxjuSIa7foAwPWGOts+6o7sgjz41/qMD9VQHEDxi6PBvK2l0MXUmqZyNpUK+T2tQaaElvw==", + "license": "MIT", + "optional": true, + "dependencies": { + "type-fest": "^0.13.1" + }, + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/serialize-error/node_modules/type-fest": { + "version": "0.13.1", + "resolved": "https://registry.npmjs.org/type-fest/-/type-fest-0.13.1.tgz", + "integrity": "sha512-34R7HTnG0XIJcBSn5XhDd7nNFPRcXYRZrBB2O2jdKqYODldSzBAqzsWoZYYvduky73toYS/ESqxPvkDf/F0XMg==", + "license": "(MIT OR CC0-1.0)", + "optional": true, + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, "node_modules/serve-static": { "version": "2.2.1", "resolved": "https://registry.npmjs.org/serve-static/-/serve-static-2.2.1.tgz", @@ -5025,6 +5884,56 @@ "integrity": "sha512-E5LDX7Wrp85Kil5bhZv46j8jOeboKq5JMmYM3gVGdGH8xFpPWXUMsNrlODCrkoxMEeNi/XZIwuRvY4XNwYMJpw==", "license": "ISC" }, + "node_modules/sharp": { + "version": "0.35.4", + "resolved": "https://registry.npmjs.org/sharp/-/sharp-0.35.4.tgz", + "integrity": "sha512-n++8XWcj+jCOr2IOl7h8LbKnGBDY4aPbmprMONBNFdn0ImXqpGVv5zliDs0V9HbmbCQLpbuo2ej9rAoOQTvMDA==", + "license": "Apache-2.0", + "optional": true, + "dependencies": { + "@img/colour": "^1.1.0", + "detect-libc": "^2.1.2", + "semver": "^7.8.5" + }, + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-darwin-arm64": "0.35.4", + "@img/sharp-darwin-x64": "0.35.4", + "@img/sharp-freebsd-wasm32": "0.35.4", + "@img/sharp-libvips-darwin-arm64": "1.3.3", + "@img/sharp-libvips-darwin-x64": "1.3.3", + "@img/sharp-libvips-linux-arm": "1.3.3", + "@img/sharp-libvips-linux-arm64": "1.3.3", + "@img/sharp-libvips-linux-ppc64": "1.3.3", + "@img/sharp-libvips-linux-riscv64": "1.3.3", + "@img/sharp-libvips-linux-s390x": "1.3.3", + "@img/sharp-libvips-linux-x64": "1.3.3", + "@img/sharp-libvips-linuxmusl-arm64": "1.3.3", + "@img/sharp-libvips-linuxmusl-x64": "1.3.3", + "@img/sharp-linux-arm": "0.35.4", + "@img/sharp-linux-arm64": "0.35.4", + "@img/sharp-linux-ppc64": "0.35.4", + "@img/sharp-linux-riscv64": "0.35.4", + "@img/sharp-linux-s390x": "0.35.4", + "@img/sharp-linux-x64": "0.35.4", + "@img/sharp-linuxmusl-arm64": "0.35.4", + "@img/sharp-linuxmusl-x64": "0.35.4", + "@img/sharp-webcontainers-wasm32": "0.35.4", + "@img/sharp-win32-arm64": "0.35.4", + "@img/sharp-win32-ia32": "0.35.4", + "@img/sharp-win32-x64": "0.35.4" + }, + "peerDependenciesMeta": { + "@types/node": { + "optional": true + } + } + }, "node_modules/shebang-command": { "version": "2.0.0", "resolved": "https://registry.npmjs.org/shebang-command/-/shebang-command-2.0.0.tgz", @@ -5218,6 +6127,13 @@ "node": ">= 10.x" } }, + "node_modules/sprintf-js": { + "version": "1.1.3", + "resolved": "https://registry.npmjs.org/sprintf-js/-/sprintf-js-1.1.3.tgz", + "integrity": "sha512-Oo+0REFV59/rz3gfJNKQiBlwfHaSESl1pcGyABQsnnIfWOFt6JNj5gCog2U6MLZ//IGYD+nA8nI+mTShREReaA==", + "license": "BSD-3-Clause", + "optional": true + }, "node_modules/ssh2": { "version": "1.17.0", "resolved": "https://registry.npmjs.org/ssh2/-/ssh2-1.17.0.tgz", @@ -5469,7 +6385,6 @@ "version": "2.8.1", "resolved": "https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz", "integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==", - "dev": true, "license": "0BSD", "optional": true }, diff --git a/package.json b/package.json index 137f397..4c3404b 100644 --- a/package.json +++ b/package.json @@ -38,7 +38,7 @@ "start:supervised": "while true; do node index.js 2>&1 | npx pino-pretty --sync; code=$?; echo \"[supervisor] lynkr exited ($code) \u2014 restarting in 3s\"; sleep 3; done", "lint": "eslint src index.js", "test": "npm run test:unit && npm run test:performance", - "test:unit": "LYNKR_KNN_DIR=/tmp/lynkr-test-knn DATABRICKS_API_KEY=test-key DATABRICKS_API_BASE=http://test.com LOG_FILE_ENABLED=false node --test test/routing.test.js test/hybrid-routing-integration.test.js test/retry-logic.test.js test/sse-transformer.test.js test/passthrough-stream.test.js test/passthrough-mode.test.js test/openrouter-error-resilience.test.js test/format-conversion.test.js test/azure-openai-config.test.js test/azure-openai-format-conversion.test.js test/azure-openai-routing.test.js test/azure-openai-streaming.test.js test/azure-openai-error-resilience.test.js test/azure-openai-integration.test.js test/openai-integration.test.js test/atlas-integration.test.js test/toon-compression.test.js test/gcf-compression.test.js test/llamacpp-integration.test.js test/resilience.test.js test/telemetry-routing.test.js test/memory/store.test.js test/memory/surprise.test.js test/memory/extractor.test.js test/memory/search.test.js test/memory/retriever.test.js test/memory/distiller.test.js test/memory/distiller-freeze.test.js test/memory/wiki.test.js test/memory/skills-cache.test.js test/memory/tencentdb-launcher.test.js test/distill.test.js test/large-payload.test.js test/prompt-cache-injection.test.js test/risk-analyzer.test.js test/interaction-block.test.js test/preflight.test.js test/token-reduction.test.js test/session-affinity.test.js test/cache-state.test.js test/cache-switch-cost.test.js test/lens-recommendations.test.js test/model-registry-cost.test.js test/output-format-guard.test.js test/tier-fallback.test.js test/wrap.test.js test/init.test.js test/tool-call-response-metadata.test.js test/degradation.test.js test/routing-telemetry-columns.test.js test/sticky-routing.test.js test/knn-ambiguous-escalate.test.js test/deescalator.test.js test/client-profiles.test.js test/strip-internal-fields.test.js test/complexity-tool-subtraction.test.js test/bandit.test.js test/routing-propensity.test.js test/reward-pipeline.test.js test/knn-cold-start.test.js test/calibration.test.js test/feedback-loop.test.js test/session-fingerprint.test.js test/side-request-guards.test.js test/verifier.test.js test/intent-score.test.js test/difficulty-classifier.test.js test/classifier-setup.test.js test/usage-stats.test.js test/loop-guard.test.js test/moonshot-model-mapping.test.js test/baidu-model-mapping.test.js test/tenant-policy-ingress-parity.test.js test/compression-budget.test.js test/gpt-utils.test.js test/dedup-observe-only.test.js test/context-window-header.test.js test/token-budget-auto.test.js test/opencode-setup.test.js test/auth-mode-first-party.test.js", + "test:unit": "LYNKR_KNN_DIR=/tmp/lynkr-test-knn DATABRICKS_API_KEY=test-key DATABRICKS_API_BASE=http://test.com LOG_FILE_ENABLED=false node --test test/routing.test.js test/hybrid-routing-integration.test.js test/retry-logic.test.js test/sse-transformer.test.js test/passthrough-stream.test.js test/passthrough-mode.test.js test/openrouter-error-resilience.test.js test/format-conversion.test.js test/azure-openai-config.test.js test/azure-openai-format-conversion.test.js test/azure-openai-routing.test.js test/azure-openai-streaming.test.js test/azure-openai-error-resilience.test.js test/azure-openai-integration.test.js test/openai-integration.test.js test/atlas-integration.test.js test/toon-compression.test.js test/gcf-compression.test.js test/llamacpp-integration.test.js test/resilience.test.js test/telemetry-routing.test.js test/memory/store.test.js test/memory/surprise.test.js test/memory/extractor.test.js test/memory/search.test.js test/memory/retriever.test.js test/memory/distiller.test.js test/memory/distiller-freeze.test.js test/memory/wiki.test.js test/memory/skills-cache.test.js test/memory/tencentdb-launcher.test.js test/distill.test.js test/large-payload.test.js test/prompt-cache-injection.test.js test/risk-analyzer.test.js test/interaction-block.test.js test/preflight.test.js test/token-reduction.test.js test/session-affinity.test.js test/cache-state.test.js test/cache-switch-cost.test.js test/lens-recommendations.test.js test/model-registry-cost.test.js test/output-format-guard.test.js test/tier-fallback.test.js test/wrap.test.js test/init.test.js test/tool-call-response-metadata.test.js test/degradation.test.js test/routing-telemetry-columns.test.js test/sticky-routing.test.js test/knn-ambiguous-escalate.test.js test/deescalator.test.js test/client-profiles.test.js test/strip-internal-fields.test.js test/complexity-tool-subtraction.test.js test/bandit.test.js test/routing-propensity.test.js test/reward-pipeline.test.js test/knn-cold-start.test.js test/calibration.test.js test/feedback-loop.test.js test/session-fingerprint.test.js test/side-request-guards.test.js test/verifier.test.js test/intent-score.test.js test/difficulty-classifier.test.js test/classifier-setup.test.js test/usage-stats.test.js test/loop-guard.test.js test/moonshot-model-mapping.test.js test/baidu-model-mapping.test.js test/tenant-policy-ingress-parity.test.js test/decide.test.js test/embeddings-degradation.test.js test/health-probe.test.js test/stuck-detector.test.js test/onnx-embedder.test.js test/ope.test.js test/hierarchical-budget.test.js test/token-rate-limit.test.js test/otel-export.test.js test/mcp-broker.test.js test/compression-budget.test.js test/gpt-utils.test.js test/dedup-observe-only.test.js test/context-window-header.test.js test/token-budget-auto.test.js test/opencode-setup.test.js test/auth-mode-first-party.test.js", "test:memory": "LYNKR_KNN_DIR=/tmp/lynkr-test-knn DATABRICKS_API_KEY=test-key DATABRICKS_API_BASE=http://test.com node --test test/memory/store.test.js test/memory/surprise.test.js test/memory/extractor.test.js test/memory/search.test.js test/memory/retriever.test.js test/memory/distiller.test.js test/memory/distiller-freeze.test.js test/memory/wiki.test.js test/memory/skills-cache.test.js test/memory/tencentdb-launcher.test.js", "test:new-features": "LYNKR_KNN_DIR=/tmp/lynkr-test-knn DATABRICKS_API_KEY=test-key DATABRICKS_API_BASE=http://test.com node --test test/passthrough-mode.test.js test/openrouter-error-resilience.test.js test/format-conversion.test.js", "test:performance": "LYNKR_KNN_DIR=/tmp/lynkr-test-knn DATABRICKS_API_KEY=test-key DATABRICKS_API_BASE=http://test.com node test/hybrid-routing-performance.test.js && DATABRICKS_API_KEY=test-key DATABRICKS_API_BASE=http://test.com node test/performance-tests.js", @@ -104,6 +104,7 @@ "pino-roll": "^4.0.0" }, "optionalDependencies": { + "@huggingface/transformers": "^4.2.0", "better-sqlite3": "^12.11.1", "dockerode": "^5.0.1", "tree-sitter": "^0.21.1", @@ -117,5 +118,9 @@ "nodemon": "^3.1.0", "pino-pretty": "^10.2.0" }, - "allowScripts": {} + "allowScripts": {}, + "overrides": { + "sharp": "^0.35.4", + "adm-zip": "^0.6.0" + } } \ No newline at end of file diff --git a/scripts/ope-report.js b/scripts/ope-report.js new file mode 100644 index 0000000..1e4dc89 --- /dev/null +++ b/scripts/ope-report.js @@ -0,0 +1,78 @@ +#!/usr/bin/env node +/** + * Off-policy evaluation report (ROUTING-NOTES §3 + §4.10.3). + * + * Scores counterfactual routing policies against logged production + * decisions — no live traffic involved. Reads routing_telemetry rows + * (propensity + candidates + quality_score, and context where the bandit + * ran) and prints IPS / SNIPS / DR / WDR estimates for each reference + * policy, alongside the logged policy's actual mean reward. + * + * Usage: + * node scripts/ope-report.js [--days 30] [--limit 50000] + * + * Reading the output: + * - "logged" is what the live policy actually achieved (ground truth). + * - A candidate policy whose WDR estimate beats "logged" — with a healthy + * effective sample size — is evidence worth acting on. A tiny ESS means + * a few high-weight rows dominate; distrust the number regardless of + * how good it looks. + * - Rows logged before the `context` column existed contribute to + * IPS/SNIPS but not to the DR regression term (drRows shows coverage). + */ + +const telemetry = require('../src/routing/telemetry'); +const { evaluatePolicy, policies } = require('../src/routing/ope'); + +function arg(name, fallback) { + const idx = process.argv.indexOf(`--${name}`); + if (idx === -1 || idx + 1 >= process.argv.length) return fallback; + const n = Number.parseInt(process.argv[idx + 1], 10); + return Number.isNaN(n) ? fallback : n; +} + +const days = arg('days', 30); +const limit = arg('limit', 50_000); + +const rows = telemetry.query({ + since: Date.now() - days * 24 * 60 * 60 * 1000, + limit, +}); + +if (rows.length === 0) { + console.log(`No telemetry rows in the last ${days} days.`); + process.exit(0); +} + +// Multi-candidate rows are where policies can actually differ; single- +// candidate rows contribute identically to every policy's estimate. +const multiCandidate = rows.filter((r) => { + try { + const c = typeof r.candidates === 'string' ? JSON.parse(r.candidates) : r.candidates; + return Array.isArray(c) && c.length > 1; + } catch { + return false; + } +}); + +console.log(`\nOff-policy evaluation — last ${days} days`); +console.log(`rows: ${rows.length} total, ${multiCandidate.length} with a real choice (>1 candidate)\n`); + +const fmt = (v) => (v == null ? ' n/a' : (v * 100).toFixed(2).padStart(6)); + +const results = []; +for (const [name, policyFn] of Object.entries(policies)) { + const r = evaluatePolicy(rows, policyFn); + results.push({ name, ...r }); +} + +const logged = results[0]?.loggedMeanReward; +console.log(`logged policy actual mean reward: ${logged == null ? 'n/a' : (logged * 100).toFixed(2)} (quality points)\n`); +console.log('policy | IPS | SNIPS | DR | WDR | ESS | usable | drRows'); +console.log('-----------------------+--------+--------+--------+--------+---------+--------+-------'); +for (const r of results) { + console.log( + `${r.name.padEnd(22)} | ${fmt(r.ips)} | ${fmt(r.snips)} | ${fmt(r.dr)} | ${fmt(r.wdr)} | ${String(r.effectiveSampleSize == null ? 'n/a' : Math.round(r.effectiveSampleSize)).padStart(7)} | ${String(r.usable).padStart(6)} | ${String(r.drRows).padStart(6)}` + ); +} +console.log('\nEstimates are quality points (0-100 scale). Prefer WDR; check ESS before trusting any gap.'); diff --git a/src/api/mcp-broker.js b/src/api/mcp-broker.js new file mode 100644 index 0000000..91109e8 --- /dev/null +++ b/src/api/mcp-broker.js @@ -0,0 +1,149 @@ +/** + * MCP broker (ROUTING-NOTES Track B / §1 gap: Lynkr was MCP-client-only). + * + * Exposes Lynkr's configured MCP servers to OTHER clients over HTTP, so one + * Lynkr install can act as the single MCP access point for a team/toolchain + * instead of every client wiring up (and holding credentials for) every + * server separately. The credential-injection property falls out of the + * existing registry design: downstream server env/credentials live in + * Lynkr's MCP config and never leave this process — callers see only tool + * names and results. + * + * Surface (all JSON): + * GET /v1/mcp/servers configured servers (id + tool count) + * GET /v1/mcp/tools aggregated tools/list across servers, + * namespaced ":" + * POST /v1/mcp/tools/call { server, tool, arguments } → tool result + * + * Security: OFF by default. Enabling requires BOTH + * LYNKR_MCP_BROKER_ENABLED=true + * LYNKR_MCP_BROKER_TOKEN= (Bearer-checked on every call) + * Enabled-without-token fails CLOSED (503 + loud log) — an exposed tool + * surface must never be reachable unauthenticated (§4.4 rule: a live + * caller's input is never silently trusted). Optional allowlist: + * LYNKR_MCP_BROKER_SERVERS=id1,id2 (default: all configured) + * + * @module api/mcp-broker + */ + +const express = require('express'); +const crypto = require('crypto'); +const logger = require('../logger'); +const mcp = require('../mcp'); + +const router = express.Router(); + +const CALL_TIMEOUT_MS = Number.parseInt(process.env.LYNKR_MCP_BROKER_CALL_TIMEOUT_MS, 10) || 60_000; + +function _enabled() { + return process.env.LYNKR_MCP_BROKER_ENABLED === 'true'; +} + +function _allowedServers() { + const raw = process.env.LYNKR_MCP_BROKER_SERVERS; + if (!raw) return null; // null = all configured servers + return new Set(raw.split(',').map((s) => s.trim()).filter(Boolean)); +} + +function _timingSafeEqual(a, b) { + const ha = crypto.createHash('sha256').update(String(a)).digest(); + const hb = crypto.createHash('sha256').update(String(b)).digest(); + return crypto.timingSafeEqual(ha, hb); +} + +function brokerAuth(req, res, next) { + if (!_enabled()) { + return res.status(404).json({ error: { type: 'not_found', message: 'MCP broker is not enabled' } }); + } + const token = process.env.LYNKR_MCP_BROKER_TOKEN; + if (!token) { + // Fail closed: enabled-but-tokenless must never serve. + logger.error('[McpBroker] LYNKR_MCP_BROKER_ENABLED=true but LYNKR_MCP_BROKER_TOKEN is unset — refusing to serve. Set a token to activate the broker.'); + return res.status(503).json({ + error: { type: 'broker_misconfigured', message: 'MCP broker enabled without LYNKR_MCP_BROKER_TOKEN; refusing to serve unauthenticated' }, + }); + } + const auth = req.headers.authorization || ''; + const presented = auth.startsWith('Bearer ') ? auth.slice(7) : null; + if (!presented || !_timingSafeEqual(presented, token)) { + return res.status(401).json({ error: { type: 'unauthorized', message: 'Invalid or missing bearer token' } }); + } + next(); +} + +function _visibleServers() { + const allow = _allowedServers(); + return mcp.listServers().filter((s) => !allow || allow.has(s.id)); +} + +async function _clientFor(serverId) { + const allow = _allowedServers(); + if (allow && !allow.has(serverId)) { + const err = new Error(`server "${serverId}" is not allowlisted for the broker`); + err.status = 403; + throw err; + } + const server = mcp.getServer(serverId); + if (!server) { + const err = new Error(`unknown MCP server "${serverId}"`); + err.status = 404; + throw err; + } + return mcp.ensureClient(serverId); +} + +router.get('/v1/mcp/servers', brokerAuth, async (_req, res) => { + const servers = _visibleServers().map((s) => ({ + id: s.id, + description: s.description ?? null, + })); + res.json({ servers }); +}); + +router.get('/v1/mcp/tools', brokerAuth, async (_req, res) => { + const results = []; + const errors = {}; + await Promise.all(_visibleServers().map(async (server) => { + try { + const client = await mcp.ensureClient(server.id); + const listed = await client.request('tools/list', {}); + for (const tool of listed?.tools ?? []) { + results.push({ + name: `${server.id}:${tool.name}`, + server: server.id, + tool: tool.name, + description: tool.description ?? null, + input_schema: tool.inputSchema ?? tool.input_schema ?? null, + }); + } + } catch (err) { + // Per-server failure is reported, not hidden — a dead server must not + // silently vanish from the tool list (§4.3: no invisible degradation). + errors[server.id] = err.message; + } + })); + res.json({ tools: results, ...(Object.keys(errors).length ? { server_errors: errors } : {}) }); +}); + +router.post('/v1/mcp/tools/call', brokerAuth, async (req, res) => { + const { server, tool, arguments: args } = req.body || {}; + if (!server || !tool) { + return res.status(400).json({ error: { type: 'invalid_request', message: 'body must include { server, tool, arguments? }' } }); + } + try { + const client = await _clientFor(server); + const result = await Promise.race([ + client.request('tools/call', { name: tool, arguments: args ?? {} }), + new Promise((_, reject) => + setTimeout(() => reject(Object.assign(new Error('tool call timed out'), { status: 504 })), CALL_TIMEOUT_MS).unref?.() + ), + ]); + res.json({ server, tool, result }); + } catch (err) { + const status = err.status ?? 502; + logger.warn({ server, tool, error: err.message }, '[McpBroker] tool call failed'); + res.status(status).json({ error: { type: 'tool_call_failed', message: err.message, server, tool } }); + } +}); + +module.exports = router; diff --git a/src/api/middleware/budget-enforcer.js b/src/api/middleware/budget-enforcer.js index 427c234..04ae282 100644 --- a/src/api/middleware/budget-enforcer.js +++ b/src/api/middleware/budget-enforcer.js @@ -21,18 +21,88 @@ function _readContext(req) { }; } +// Default expected-output tokens when the caller didn't set max_tokens — +// same shape as LiteLLM's TPM floor: the estimate is soft admission control, +// trued up by the post-response recordSpend(), not a hard ceiling. +const DEFAULT_OUTPUT_TOKENS_ESTIMATE = 1024; + +// Cached blended per-1k price across the configured tier models. Routing +// hasn't picked a model yet at middleware time, so the estimate prices at +// the average of what this install could plausibly serve. Refreshed every +// 60s (tier config can hot-reload). +let _priceCache = { at: 0, inputPer1k: 0, outputPer1k: 0 }; +const PRICE_CACHE_TTL_MS = 60_000; + +function _blendedTierPricing() { + const now = Date.now(); + if (now - _priceCache.at < PRICE_CACHE_TTL_MS) return _priceCache; + try { + const { getModelTierSelector } = require('../../routing/model-tiers'); + const { getModelRegistrySync } = require('../../routing/model-registry'); + const registry = getModelRegistrySync && getModelRegistrySync(); + const models = getModelTierSelector().getAllConfiguredModels(); + let inputSum = 0; + let outputSum = 0; + let priced = 0; + for (const m of models) { + const cost = registry?.getCost?.(m.model); + if (!cost || cost.unknown) continue; + inputSum += cost.input ?? 0; + outputSum += cost.output ?? 0; + priced += 1; + } + _priceCache = { + at: now, + inputPer1k: priced > 0 ? inputSum / priced : 0, + outputPer1k: priced > 0 ? outputSum / priced : 0, + }; + } catch { + _priceCache = { at: now, inputPer1k: 0, outputPer1k: 0 }; + } + return _priceCache; +} + +/** + * Pre-flight cost estimate for a request that hasn't been routed yet: + * estimated input tokens (4-chars≈1-token over system+tools+messages) plus + * expected output (caller's max_tokens, else a floor), priced at the blended + * average of the configured tier models. Falls back to a $0.01 nominal gate + * when nothing is priceable — never LESS strict than the old behavior. + * + * @param {object} body — request payload + * @returns {number} estimated USD cost + */ +function estimateRequestCost(body) { + try { + const { countPayloadTokens } = require('../../utils/tokens'); + const inputTokens = countPayloadTokens(body || {}).total || 0; + const outputTokens = Math.min( + typeof body?.max_tokens === 'number' && body.max_tokens > 0 + ? body.max_tokens + : DEFAULT_OUTPUT_TOKENS_ESTIMATE, + 32_000, + ); + const price = _blendedTierPricing(); + const est = (inputTokens / 1000) * price.inputPer1k + + (outputTokens / 1000) * price.outputPer1k; + // Floor at the old nominal $0.01 so free/unpriced configs still gate + // exhausted accounts exactly as before. + return Math.max(0.01, est); + } catch { + return 0.01; + } +} + /** - * Express middleware. Estimates request cost via cost-optimizer and rejects - * if the budget is already exceeded. Records spend after the response. + * Express middleware. Estimates request cost from the payload's token count + * and blended tier pricing, and rejects if the budget would be exceeded. + * Actual spend is recorded post-response (estimate → true-up pattern). */ function budgetEnforcer(req, res, next) { if (process.env.LYNKR_BUDGET_ENFORCER === 'false') return next(); const context = _readContext(req); - // Cheap pre-check at $0; we use the request to record actual spend. - // The actual ceiling check happens with an estimated $0.01 "minimum" so - // exhausted accounts get rejected before we even route. const budget = getHierarchicalBudget(); - const check = budget.check(context, 0.01); + const check = budget.check(context, estimateRequestCost(req.body)); if (!check.ok) { logger.warn({ exceeded: check.exceeded }, '[BudgetEnforcer] Budget exceeded'); return res.status(429).json({ @@ -57,4 +127,4 @@ function recordSpend(context, amount) { getHierarchicalBudget().record(context, amount); } -module.exports = { budgetEnforcer, recordSpend }; +module.exports = { budgetEnforcer, recordSpend, estimateRequestCost }; diff --git a/src/api/middleware/budget.js b/src/api/middleware/budget.js index e54ecb7..e8ead37 100644 --- a/src/api/middleware/budget.js +++ b/src/api/middleware/budget.js @@ -30,6 +30,36 @@ function budgetMiddleware(req, res, next) { }); } + // Token-aware (TPM) rate limiting — off unless LYNKR_TPM_LIMIT is set. + // Pre-flight gates on the window's actual consumption plus this request's + // cheap estimate; the true-up lands in the res 'finish' handler below. + try { + const { countPayloadTokens } = require('../../utils/tokens'); + const estInput = countPayloadTokens(req.body || {}).total || 0; + const estOutput = typeof req.body?.max_tokens === 'number' && req.body.max_tokens > 0 + ? req.body.max_tokens + : 1024; + const tokenCheck = budgetManager.checkTokenRate(userId, estInput + estOutput); + if (!tokenCheck.allowed) { + logger.warn({ + userId, + limit: tokenCheck.limit, + current: tokenCheck.current, + estimated: tokenCheck.estimated, + }, 'Token rate limit (TPM) exceeded'); + return res.status(429).json({ + error: 'rate_limit_exceeded', + message: `Token rate limit exceeded: ${tokenCheck.limit} tokens per minute`, + limit: tokenCheck.limit, + current: tokenCheck.current, + resetInMs: tokenCheck.resetInMs, + retryAfter: Math.ceil(tokenCheck.resetInMs / 1000), + }); + } + } catch (err) { + logger.debug({ err: err.message }, 'TPM check failed — allowing request'); + } + // Check budget const budgetCheck = budgetManager.checkBudget(userId); if (!budgetCheck.allowed) { @@ -68,14 +98,18 @@ function budgetMiddleware(req, res, next) { try { const usage = res.locals.usage; if (!usage) return; + const tokensInput = usage.prompt_tokens || usage.input_tokens || 0; + const tokensOutput = usage.completion_tokens || usage.output_tokens || 0; budgetManager.recordUsage(userId, req.session?.id || null, { - tokensInput: usage.prompt_tokens || usage.input_tokens || 0, - tokensOutput: usage.completion_tokens || usage.output_tokens || 0, + tokensInput, + tokensOutput, costUsd: usage.cost_usd || 0, model: usage.model || null, endpoint: req.path, latencyMs: Date.now() - req.budgetInfo.startTime, }); + // TPM true-up with actual consumption (no-op unless LYNKR_TPM_LIMIT set). + budgetManager.recordTokenUsage(userId, tokensInput + tokensOutput); } catch (err) { logger.warn({ err: err.message }, 'Failed to record usage after response'); } diff --git a/src/budget/index.js b/src/budget/index.js index 5e31968..dba7ef0 100644 --- a/src/budget/index.js +++ b/src/budget/index.js @@ -75,6 +75,12 @@ class BudgetManager { minute_window_start INTEGER, hour_window_start INTEGER ); + + CREATE TABLE IF NOT EXISTS token_rate ( + user_id TEXT PRIMARY KEY, + tokens_minute INTEGER NOT NULL DEFAULT 0, + minute_window_start INTEGER NOT NULL + ); `); this.stmts = { @@ -114,6 +120,86 @@ class BudgetManager { }; } + /** + * Token-aware (TPM) rate limiting (ROUTING-NOTES §1 gap: request-count + * only). Off unless LYNKR_TPM_LIMIT is set — same "off unless configured" + * convention as the loop guard. + * + * Estimate → true-up pattern (the one every serious gateway converged on): + * the pre-flight check gates on the window's ACTUAL consumption plus the + * current request's cheap estimate; real usage lands in the window + * post-response via recordTokenUsage(). Soft admission control, not a hard + * ceiling — concurrent in-flight requests can overshoot by roughly one + * request's worth, by design. + * + * @param {string} userId + * @param {number} estimatedTokens — cheap pre-flight estimate for THIS request + * @returns {{allowed: boolean, reason?: string, limit?: number, current?: number, resetInMs?: number}} + */ + checkTokenRate(userId, estimatedTokens = 0) { + if (!this.enabled) return { allowed: true }; + const limit = Number.parseInt(process.env.LYNKR_TPM_LIMIT, 10); + if (!limit || limit <= 0 || Number.isNaN(limit)) return { allowed: true }; + + const now = Date.now(); + const minuteWindow = 60 * 1000; + const row = this.db.prepare('SELECT * FROM token_rate WHERE user_id = ?').get(userId); + let tokensMinute = row?.tokens_minute ?? 0; + let windowStart = row?.minute_window_start ?? now; + if (now - windowStart >= minuteWindow) { + tokensMinute = 0; + windowStart = now; + this.db.prepare(` + INSERT INTO token_rate (user_id, tokens_minute, minute_window_start) + VALUES (?, 0, ?) + ON CONFLICT(user_id) DO UPDATE SET tokens_minute = 0, minute_window_start = excluded.minute_window_start + `).run(userId, windowStart); + } + + if (tokensMinute + estimatedTokens > limit) { + return { + allowed: false, + reason: 'token_rate_limit_minute', + limit, + current: tokensMinute, + estimated: estimatedTokens, + resetInMs: minuteWindow - (now - windowStart), + }; + } + return { allowed: true }; + } + + /** + * True-up: add ACTUAL token consumption to the user's TPM window after the + * response completes. No-op when TPM limiting is disabled. + * @param {string} userId + * @param {number} totalTokens — actual input+output tokens consumed + */ + recordTokenUsage(userId, totalTokens) { + if (!this.enabled || !totalTokens || totalTokens <= 0) return; + const limit = Number.parseInt(process.env.LYNKR_TPM_LIMIT, 10); + if (!limit || limit <= 0 || Number.isNaN(limit)) return; + const now = Date.now(); + const minuteWindow = 60 * 1000; + try { + const row = this.db.prepare('SELECT * FROM token_rate WHERE user_id = ?').get(userId); + const stale = !row || now - row.minute_window_start >= minuteWindow; + this.db.prepare(` + INSERT INTO token_rate (user_id, tokens_minute, minute_window_start) + VALUES (@user_id, @tokens, @window_start) + ON CONFLICT(user_id) DO UPDATE SET + tokens_minute = ${stale ? '@tokens' : 'tokens_minute + @tokens'}, + minute_window_start = @window_start + `).run({ + user_id: userId, + tokens: totalTokens, + window_start: stale ? now : row.minute_window_start, + }); + } catch (err) { + logger.debug({ err: err.message }, '[Budget] token usage record failed'); + } + } + checkRateLimit(userId) { if (!this.enabled) return { allowed: true }; diff --git a/src/cache/embeddings.js b/src/cache/embeddings.js index 4edb974..a11afe1 100644 --- a/src/cache/embeddings.js +++ b/src/cache/embeddings.js @@ -115,55 +115,136 @@ function simpleHash(str) { return hash; } -// Track if embedding provider is available +// Embedding provider availability state. +// +// The hash fallback is NOT a semantic embedding — when it's active, the +// semantic cache and kNN router are effectively disabled (hash vectors are +// 384-dim vs the providers' 768-dim, so they never match real entries; the +// kNN index rejects them on dimension). Historically this degradation was a +// permanent latch (one transient provider failure disabled semantic matching +// until process restart) and logged only at debug level — i.e. invisible. +// The same failure shape elsewhere in the industry silently misrouted ~20% +// of a production cluster's spend before anyone noticed, so this state is +// now: retried on a cooldown, logged loudly on every transition, counted, +// and exposed via getEmbeddingStatus() for /metrics. let embeddingProviderAvailable = null; +let degradedSince = null; +let lastProviderAttempt = 0; +let fallbackCount = 0; +let lastProviderError = null; + +// After a provider failure, wait this long before trying it again (instead +// of latching to the fallback forever). Env-tunable for tests/impatience. +const RETRY_COOLDOWN_MS = Number.parseInt(process.env.LYNKR_EMBEDDINGS_RETRY_COOLDOWN_MS, 10) || 60_000; + +// Strict mode: throw instead of silently degrading to hash embeddings. +// For deployments that prefer fail-loud (no semantic cache is better than a +// silently fake one). Default stays fail-soft to match Lynkr's philosophy. +const STRICT = process.env.LYNKR_EMBEDDINGS_STRICT === 'true'; + +function _noteFallback(providerName, err) { + fallbackCount += 1; + lastProviderError = err?.message || String(err); + if (embeddingProviderAvailable !== false) { + embeddingProviderAvailable = false; + degradedSince = Date.now(); + logger.warn({ + provider: providerName, + error: lastProviderError, + retryCooldownMs: RETRY_COOLDOWN_MS, + }, '[Embeddings] Provider unreachable — DEGRADED to non-semantic hash embeddings. Semantic cache and kNN routing are effectively disabled until the provider recovers.'); + } +} + +function _noteRecovery(providerName) { + if (embeddingProviderAvailable === false) { + logger.info({ + provider: providerName, + degradedForMs: degradedSince ? Date.now() - degradedSince : null, + fallbacksServed: fallbackCount, + }, '[Embeddings] Provider recovered — semantic embeddings restored'); + } + embeddingProviderAvailable = true; + degradedSince = null; +} + +function _wrapProvider(providerName, providerFn) { + return async (text) => { + // While degraded, only re-attempt the provider after the cooldown; serve + // the fallback in between so a dead provider doesn't add per-request + // connect timeouts to the hot path. + if (embeddingProviderAvailable === false + && Date.now() - lastProviderAttempt < RETRY_COOLDOWN_MS) { + if (STRICT) throw new Error(`Embedding provider ${providerName} degraded: ${lastProviderError}`); + fallbackCount += 1; + return generateHashEmbedding(text); + } + lastProviderAttempt = Date.now(); + try { + const result = await providerFn(text); + _noteRecovery(providerName); + return result; + } catch (err) { + _noteFallback(providerName, err); + if (STRICT) throw err; + return generateHashEmbedding(text); + } + }; +} /** * Get the appropriate embedding function based on config * @returns {Function} - Embedding generation function */ function getEmbeddingFunction() { - // If we already know embedding provider isn't available, use fallback - if (embeddingProviderAvailable === false) { - return (text) => Promise.resolve(generateHashEmbedding(text)); - } - const provider = config.modelProvider?.type || 'databricks'; + // In-process ONNX embedder (opt-in): no external embedding server on the + // hot path at all. Same underlying model as the Ollama default + // (nomic-embed-text, 768-dim), so existing kNN/cache vectors stay valid. + // Wrapped in the same degradation machinery — a failed model load logs + // loudly, serves the hash fallback, and retries after the cooldown. + if (process.env.LYNKR_EMBEDDINGS_PROVIDER === 'onnx') { + const { generateOnnxEmbedding, isOnnxAvailable } = require('./onnx-embedder'); + if (isOnnxAvailable()) { + return _wrapProvider('onnx', generateOnnxEmbedding); + } + logger.warn('[Embeddings] LYNKR_EMBEDDINGS_PROVIDER=onnx but @huggingface/transformers is not installed (optionalDependency) — falling through to the configured network provider'); + } + // Check if we have a local embedding provider configured if (config.ollama?.embeddingsEndpoint || provider === 'ollama') { - return async (text) => { - try { - const result = await generateOllamaEmbedding(text); - embeddingProviderAvailable = true; - return result; - } catch (err) { - logger.debug({ error: err.message }, 'Ollama embedding failed, using hash fallback'); - embeddingProviderAvailable = false; - return generateHashEmbedding(text); - } - }; + return _wrapProvider('ollama', generateOllamaEmbedding); } if (config.llamacpp?.embeddingsEndpoint || provider === 'llamacpp') { - return async (text) => { - try { - const result = await generateLlamaCppEmbedding(text); - embeddingProviderAvailable = true; - return result; - } catch (err) { - logger.debug({ error: err.message }, 'LlamaCpp embedding failed, using hash fallback'); - embeddingProviderAvailable = false; - return generateHashEmbedding(text); - } - }; - } - - // Fallback to hash-based embeddings - logger.debug('No embedding provider configured, using hash-based fallback'); + return _wrapProvider('llamacpp', generateLlamaCppEmbedding); + } + + // No provider configured at all — hash fallback is the deliberate mode, + // not a degradation. Warn once so the operator knows semantic matching is + // approximate, then stay quiet. + if (embeddingProviderAvailable === null) { + embeddingProviderAvailable = false; + logger.warn('[Embeddings] No embedding provider configured — using non-semantic hash embeddings. Semantic cache matches will be approximate; configure an Ollama/llama.cpp embeddings endpoint for real semantic matching.'); + } return (text) => Promise.resolve(generateHashEmbedding(text)); } +/** + * Current embedding subsystem status, for /metrics and health surfaces. + * @returns {{ providerAvailable: boolean|null, degradedSince: number|null, + * fallbackCount: number, lastProviderError: string|null }} + */ +function getEmbeddingStatus() { + return { + providerAvailable: embeddingProviderAvailable, + degradedSince, + fallbackCount, + lastProviderError, + }; +} + /** * Generate embedding for text * @param {string} text - Text to embed @@ -178,12 +259,16 @@ async function generateEmbedding(text) { const maxLength = 8000; const truncated = text.length > maxLength ? text.substring(0, maxLength) : text; + const embedFn = getEmbeddingFunction(); + if (STRICT) return embedFn(truncated); try { - const embedFn = getEmbeddingFunction(); return await embedFn(truncated); } catch (err) { - // Final fallback to hash embeddings if everything else fails - logger.debug({ error: err.message }, 'Embedding generation failed, using hash fallback'); + // Final fallback to hash embeddings if everything else fails. The + // provider wrapper already logged the degradation transition loudly; + // this catch only covers unexpected non-provider errors. + logger.warn({ error: err.message }, '[Embeddings] Embedding generation failed unexpectedly, serving hash fallback'); + fallbackCount += 1; return generateHashEmbedding(truncated); } } @@ -193,6 +278,10 @@ async function generateEmbedding(text) { */ function resetEmbeddingProvider() { embeddingProviderAvailable = null; + degradedSince = null; + lastProviderAttempt = 0; + fallbackCount = 0; + lastProviderError = null; } /** @@ -231,5 +320,6 @@ module.exports = { generateHashEmbedding, cosineSimilarity, getEmbeddingFunction, + getEmbeddingStatus, resetEmbeddingProvider, }; diff --git a/src/cache/onnx-embedder.js b/src/cache/onnx-embedder.js new file mode 100644 index 0000000..1eae9df --- /dev/null +++ b/src/cache/onnx-embedder.js @@ -0,0 +1,92 @@ +/** + * In-process ONNX embedder (ROUTING-NOTES §4.10.2). + * + * Runs the embedding model inside the Lynkr process via transformers.js + * (@huggingface/transformers, an optionalDependency), removing the external + * Ollama/llama.cpp server from the semantic-cache and kNN hot path — no + * network hop, no "is the server up" failure mode, no contention with + * completion traffic on the same local runtime. + * + * Model choice is deliberate: the ONNX build of the SAME model the Ollama + * path defaults to (nomic-embed-text, 768-dim). Same model = same embedding + * space = existing kNN index entries and semantic-cache vectors stay valid. + * (INT8 quantization introduces small numeric differences vs Ollama's + * serving precision, but cosine similarity is preserved far above the 0.92 + * cache threshold.) + * + * Weights are downloaded once from the HuggingFace hub into + * ~/.lynkr/models (override: LYNKR_ONNX_CACHE_DIR) and verified by + * transformers.js's own integrity checks. Nothing is bundled in the npm + * package. + * + * Opt-in via LYNKR_EMBEDDINGS_PROVIDER=onnx. If the optional dependency is + * missing or the model fails to load, the caller's degradation machinery + * (cache/embeddings.js) takes over — loud, observable, recoverable. + * + * @module cache/onnx-embedder + */ + +const os = require('os'); +const path = require('path'); +const logger = require('../logger'); + +const MODEL_ID = process.env.LYNKR_ONNX_EMBEDDING_MODEL || 'Xenova/nomic-embed-text-v1'; +const CACHE_DIR = process.env.LYNKR_ONNX_CACHE_DIR + || path.join(os.homedir(), '.lynkr', 'models'); +// q8 keeps the download ~25–35MB and inference CPU-friendly. +const DTYPE = process.env.LYNKR_ONNX_DTYPE || 'q8'; + +/** @type {Promise|null} memoized pipeline load (single flight) */ +let pipelinePromise = null; + +async function _loadPipeline() { + // Lazy, inside the function: @huggingface/transformers is an + // optionalDependency and an ESM-only package — import() from CJS. + const { pipeline, env } = await import('@huggingface/transformers'); + env.cacheDir = CACHE_DIR; + const started = Date.now(); + logger.info({ model: MODEL_ID, cacheDir: CACHE_DIR, dtype: DTYPE }, + '[OnnxEmbedder] Loading embedding model (first run downloads weights)'); + const extractor = await pipeline('feature-extraction', MODEL_ID, { dtype: DTYPE }); + logger.info({ model: MODEL_ID, loadMs: Date.now() - started }, + '[OnnxEmbedder] Embedding model ready'); + return extractor; +} + +/** + * Generate an embedding fully in-process. + * @param {string} text + * @returns {Promise} mean-pooled, L2-normalized vector (768-dim + * for the default model) + */ +async function generateOnnxEmbedding(text) { + if (!pipelinePromise) { + pipelinePromise = _loadPipeline().catch((err) => { + // Reset so a later call can retry (e.g. transient download failure) — + // a failed load must not latch permanently, same principle as the + // provider degradation fix in cache/embeddings.js. + pipelinePromise = null; + throw err; + }); + } + const extractor = await pipelinePromise; + const output = await extractor(text, { pooling: 'mean', normalize: true }); + return Array.from(output.data); +} + +/** True when the optional dependency is installed (cheap resolve check). */ +function isOnnxAvailable() { + try { + require.resolve('@huggingface/transformers'); + return true; + } catch { + return false; + } +} + +/** Test helper — drop the memoized pipeline. */ +function _resetPipeline() { + pipelinePromise = null; +} + +module.exports = { generateOnnxEmbedding, isOnnxAvailable, _resetPipeline, MODEL_ID }; diff --git a/src/clients/databricks.js b/src/clients/databricks.js index c867fd6..99f0a53 100644 --- a/src/clients/databricks.js +++ b/src/clients/databricks.js @@ -3405,6 +3405,7 @@ async function invokeModel(body, options = {}) { base_tier: routingResult.base_tier ?? null, escalation_source: routingResult.escalation_source ?? null, propensity: routingResult.propensity ?? null, + context: routingResult._banditContext ?? null, candidates: routingResult.candidates ?? null, pinned: routingResult.pinned ? 1 : 0, switch_reason: routingResult.switch_reason ?? null, @@ -3639,6 +3640,7 @@ async function invokeModel(body, options = {}) { base_tier: routingResult.base_tier ?? null, escalation_source: routingResult.escalation_source ?? null, propensity: routingResult.propensity ?? null, + context: routingResult._banditContext ?? null, candidates: routingResult.candidates ?? null, pinned: routingResult.pinned ? 1 : 0, switch_reason: routingResult.switch_reason ?? null, @@ -3751,6 +3753,7 @@ async function invokeModel(body, options = {}) { base_tier: routingResult.base_tier ?? null, escalation_source: routingResult.escalation_source ?? null, propensity: routingResult.propensity ?? null, + context: routingResult._banditContext ?? null, candidates: routingResult.candidates ?? null, pinned: routingResult.pinned ? 1 : 0, switch_reason: routingResult.switch_reason ?? null, @@ -3847,6 +3850,7 @@ async function invokeModel(body, options = {}) { base_tier: routingResult.base_tier ?? null, escalation_source: routingResult.escalation_source ?? null, propensity: routingResult.propensity ?? null, + context: routingResult._banditContext ?? null, candidates: routingResult.candidates ?? null, pinned: routingResult.pinned ? 1 : 0, switch_reason: routingResult.switch_reason ?? null, @@ -3907,6 +3911,7 @@ async function invokeModel(body, options = {}) { base_tier: routingResult.base_tier ?? null, escalation_source: routingResult.escalation_source ?? null, propensity: routingResult.propensity ?? null, + context: routingResult._banditContext ?? null, candidates: routingResult.candidates ?? null, pinned: routingResult.pinned ? 1 : 0, switch_reason: routingResult.switch_reason ?? null, diff --git a/src/clients/health-probe.js b/src/clients/health-probe.js new file mode 100644 index 0000000..b168c71 --- /dev/null +++ b/src/clients/health-probe.js @@ -0,0 +1,157 @@ +/** + * Synthetic circuit-breaker health probe (ROUTING-NOTES §1 gap audit). + * + * Problem: cockatiel's halfOpenAfter is a pure timer — after it elapses, the + * NEXT LIVE USER REQUEST is the recovery probe. A user pays the latency/cost + * of testing a dead provider, and a provider with no traffic never recovers. + * + * Fix (the pattern LiteLLM ships as health-check-driven routing): a + * background loop that, for every breaker currently OPEN or HALF_OPEN, runs + * a cheap synthetic request *through the breaker* (`breaker.execute(probe)`) + * on a schedule. A success closes the circuit via cockatiel's normal + * half-open transition; a failure re-opens it. Healthy breakers are never + * probed — the loop is zero-cost when everything is up. + * + * Probes are registered per provider name (the breaker registry key). + * Built-in probes cover the local providers (ollama / llamacpp / lmstudio), + * whose cheap GET endpoints are well-known, free, and exactly where the + * dead-upstream-hang problem historically lived. Cloud providers can be + * added via registerHealthProbe(name, fn); without one, their recovery + * stays as before (next live request) — no behavior regression. + */ + +const config = require('../config'); +const logger = require('../logger'); +const { getCockatielRegistry } = require('./resilience'); + +const PROBE_INTERVAL_MS = Number.parseInt(process.env.LYNKR_HEALTH_PROBE_INTERVAL_MS, 10) || 30_000; +const PROBE_TIMEOUT_MS = Number.parseInt(process.env.LYNKR_HEALTH_PROBE_TIMEOUT_MS, 10) || 5_000; +const ENABLED = process.env.LYNKR_HEALTH_PROBE_ENABLED !== 'false'; + +/** @type {Map Promise>} providerName → probe fn (throws on unhealthy) */ +const probes = new Map(); + +async function _cheapGet(url, headers = {}) { + const response = await fetch(url, { + method: 'GET', + headers, + signal: AbortSignal.timeout(PROBE_TIMEOUT_MS), + }); + if (!response.ok) { + throw new Error(`health probe ${url} → ${response.status}`); + } + // Drain the (small) body so the socket is released cleanly. + await response.arrayBuffer().catch(() => {}); +} + +function _registerBuiltins() { + if (config.ollama?.endpoint) { + probes.set('ollama', () => _cheapGet(`${config.ollama.endpoint}/api/tags`)); + } + if (config.llamacpp?.endpoint) { + const headers = config.llamacpp.apiKey + ? { Authorization: `Bearer ${config.llamacpp.apiKey}` } + : {}; + probes.set('llamacpp', () => _cheapGet(`${config.llamacpp.endpoint}/v1/models`, headers)); + } + if (config.lmstudio?.endpoint) { + const headers = config.lmstudio.apiKey + ? { Authorization: `Bearer ${config.lmstudio.apiKey}` } + : {}; + probes.set('lmstudio', () => _cheapGet(`${config.lmstudio.endpoint}/v1/models`, headers)); + } +} + +/** + * Register (or override) a synthetic probe for a provider. The function must + * resolve when the provider is healthy and throw/reject when it is not. + * @param {string} providerName — must match the circuit breaker registry key + * @param {() => Promise} probeFn + */ +function registerHealthProbe(providerName, probeFn) { + probes.set(providerName, probeFn); +} + +class HealthProber { + constructor() { + this.timer = null; + this.stats = { sweeps: 0, probesSent: 0, recoveries: 0, failures: 0, lastSweepAt: null }; + } + + start() { + if (this.timer || !ENABLED) return; + _registerBuiltins(); + this.timer = setInterval(() => { + this.sweep().catch((err) => { + logger.debug({ error: err.message }, '[HealthProbe] sweep error'); + }); + }, PROBE_INTERVAL_MS); + // Never keep the process alive just to probe. + this.timer.unref?.(); + logger.info( + { intervalMs: PROBE_INTERVAL_MS, providers: [...probes.keys()] }, + '[HealthProbe] Synthetic circuit-breaker probing started' + ); + } + + stop() { + if (this.timer) { + clearInterval(this.timer); + this.timer = null; + } + } + + /** + * One probing pass: probe every registered provider whose breaker is not + * closed. Runs probes through the breaker itself so state transitions use + * cockatiel's own half-open machinery. + */ + async sweep() { + this.stats.sweeps += 1; + this.stats.lastSweepAt = Date.now(); + const registry = getCockatielRegistry(); + const work = []; + for (const [providerName, probeFn] of probes) { + const breaker = registry.breakers.get(providerName); + if (!breaker || breaker.state === 'CLOSED') continue; + work.push( + (async () => { + this.stats.probesSent += 1; + try { + await breaker.execute(probeFn); + this.stats.recoveries += 1; + logger.info({ provider: providerName }, '[HealthProbe] Provider recovered via synthetic probe — circuit closed without a live request paying for the test'); + } catch (err) { + // Expected while the provider is still down (or the circuit is + // OPEN and not yet past halfOpenAfter, where execute rejects + // immediately without dialing). Stay quiet; the breaker's own + // state logging covers transitions. + this.stats.failures += 1; + logger.debug({ provider: providerName, error: err.message }, '[HealthProbe] probe failed'); + } + })() + ); + } + await Promise.all(work); + } + + getStatus() { + return { + enabled: ENABLED, + running: !!this.timer, + intervalMs: PROBE_INTERVAL_MS, + registeredProviders: [...probes.keys()], + ...this.stats, + }; + } +} + +let prober = null; + +/** @returns {HealthProber} singleton */ +function getHealthProber() { + if (!prober) prober = new HealthProber(); + return prober; +} + +module.exports = { getHealthProber, registerHealthProbe }; diff --git a/src/observability/metrics.js b/src/observability/metrics.js index 5253366..b99da4c 100644 --- a/src/observability/metrics.js +++ b/src/observability/metrics.js @@ -375,6 +375,15 @@ class MetricsCollector { metric("http_request_duration_ms", "summary", "HTTP request latency in ms", metrics.latency_ms.p95, { quantile: "0.95" }); metric("http_request_duration_ms", "summary", "HTTP request latency in ms", metrics.latency_ms.p99, { quantile: "0.99" }); + // OTel GenAI semantic-convention aliases (dots → underscores, per + // Prometheus conversion rules). Same underlying counters as above, + // named so semconv-aware dashboards/tooling find them. + metric("gen_ai_client_token_usage_total", "counter", "GenAI tokens consumed (OTel semconv)", metrics.tokens_input_total, { gen_ai_token_type: "input" }); + metric("gen_ai_client_token_usage_total", "counter", "GenAI tokens consumed (OTel semconv)", metrics.tokens_output_total, { gen_ai_token_type: "output" }); + metric("gen_ai_client_operation_duration_ms", "summary", "GenAI operation duration (OTel semconv)", metrics.latency_ms.median, { quantile: "0.5" }); + metric("gen_ai_client_operation_duration_ms", "summary", "GenAI operation duration (OTel semconv)", metrics.latency_ms.p95, { quantile: "0.95" }); + metric("gen_ai_client_operation_duration_ms", "summary", "GenAI operation duration (OTel semconv)", metrics.latency_ms.p99, { quantile: "0.99" }); + return lines.join("\n"); } diff --git a/src/observability/otel.js b/src/observability/otel.js new file mode 100644 index 0000000..b1c8f47 --- /dev/null +++ b/src/observability/otel.js @@ -0,0 +1,180 @@ +/** + * OpenTelemetry GenAI metrics export (ROUTING-NOTES §1 gap audit). + * + * Lynkr already tracks every number the OTel GenAI semantic conventions + * care about — this module gives them semconv-aligned names and an export + * path, WITHOUT adding the @opentelemetry/* dependency tree: OTLP/HTTP is + * just JSON over POST, and the payload below follows the OTLP 1.x + * ExportMetricsServiceRequest shape that any collector accepts. + * + * Enabled when OTEL_EXPORTER_OTLP_ENDPOINT (the standard env var) or + * LYNKR_OTEL_ENDPOINT is set. Pushes every LYNKR_OTEL_EXPORT_INTERVAL_MS + * (default 60s) to `/v1/metrics`. Optional headers via the + * standard OTEL_EXPORTER_OTLP_HEADERS ("key=value,key2=value2"). + * + * Emitted metrics (GenAI semconv where one exists, lynkr.* where not): + * gen_ai.client.token.usage sum, attr gen_ai.token.type=input|output + * gen_ai.client.operation.duration gauge (p50/p95/p99, unit ms) + * lynkr.cost.usd sum + * lynkr.requests sum (+ lynkr.request.errors) + * + * @module observability/otel + */ + +const logger = require('../logger'); +const { getMetricsCollector } = require('./metrics'); + +const EXPORT_INTERVAL_MS = Number.parseInt(process.env.LYNKR_OTEL_EXPORT_INTERVAL_MS, 10) || 60_000; + +function _endpoint() { + const raw = process.env.OTEL_EXPORTER_OTLP_ENDPOINT || process.env.LYNKR_OTEL_ENDPOINT || null; + if (!raw) return null; + return raw.replace(/\/+$/, ''); +} + +function _headers() { + const headers = { 'Content-Type': 'application/json' }; + const raw = process.env.OTEL_EXPORTER_OTLP_HEADERS; + if (raw) { + for (const pair of raw.split(',')) { + const idx = pair.indexOf('='); + if (idx > 0) headers[pair.slice(0, idx).trim()] = pair.slice(idx + 1).trim(); + } + } + return headers; +} + +function _sumPoint(value, attributes, startNanos, nowNanos) { + return { + attributes, + startTimeUnixNano: startNanos, + timeUnixNano: nowNanos, + asDouble: value, + }; +} + +/** + * Build an OTLP/HTTP ExportMetricsServiceRequest from the collector's + * current snapshot. Exposed for tests. + */ +function buildOtlpPayload(metrics, { startTimeMs, nowMs }) { + const startNanos = String(startTimeMs * 1e6); + const nowNanos = String(nowMs * 1e6); + const attr = (key, value) => ({ key, value: { stringValue: value } }); + + const sums = (name, unit, points) => ({ + name, + unit, + sum: { + aggregationTemporality: 2, // CUMULATIVE + isMonotonic: true, + dataPoints: points, + }, + }); + + const gauge = (name, unit, points) => ({ name, unit, gauge: { dataPoints: points } }); + + return { + resourceMetrics: [{ + resource: { + attributes: [ + attr('service.name', 'lynkr'), + attr('service.version', process.env.npm_package_version || 'unknown'), + ], + }, + scopeMetrics: [{ + scope: { name: 'lynkr.observability', version: '1' }, + metrics: [ + sums('gen_ai.client.token.usage', '{token}', [ + _sumPoint(metrics.tokens_input_total, [attr('gen_ai.token.type', 'input')], startNanos, nowNanos), + _sumPoint(metrics.tokens_output_total, [attr('gen_ai.token.type', 'output')], startNanos, nowNanos), + ]), + gauge('gen_ai.client.operation.duration', 'ms', [ + { attributes: [attr('quantile', '0.5')], timeUnixNano: nowNanos, asDouble: metrics.latency_ms.median }, + { attributes: [attr('quantile', '0.95')], timeUnixNano: nowNanos, asDouble: metrics.latency_ms.p95 }, + { attributes: [attr('quantile', '0.99')], timeUnixNano: nowNanos, asDouble: metrics.latency_ms.p99 }, + ]), + sums('lynkr.cost.usd', 'usd', [ + _sumPoint(metrics.cost_usd_total, [], startNanos, nowNanos), + ]), + sums('lynkr.requests', '{request}', [ + _sumPoint(metrics.requests_total, [], startNanos, nowNanos), + ]), + sums('lynkr.request.errors', '{request}', [ + _sumPoint(metrics.requests_errors_total, [], startNanos, nowNanos), + ]), + ], + }], + }], + }; +} + +class OtelExporter { + constructor() { + this.timer = null; + this.startTimeMs = Date.now(); + this.stats = { exports: 0, failures: 0, lastExportAt: null }; + this._warnedOnce = false; + } + + start() { + const endpoint = _endpoint(); + if (!endpoint || this.timer) return; + this.timer = setInterval(() => { + this.exportOnce().catch(() => {}); + }, EXPORT_INTERVAL_MS); + this.timer.unref?.(); + logger.info({ endpoint, intervalMs: EXPORT_INTERVAL_MS }, '[Otel] GenAI metrics export started'); + } + + stop() { + if (this.timer) { + clearInterval(this.timer); + this.timer = null; + } + } + + async exportOnce() { + const endpoint = _endpoint(); + if (!endpoint) return false; + const metrics = getMetricsCollector().getMetrics(); + const payload = buildOtlpPayload(metrics, { startTimeMs: this.startTimeMs, nowMs: Date.now() }); + try { + const response = await fetch(`${endpoint}/v1/metrics`, { + method: 'POST', + headers: _headers(), + body: JSON.stringify(payload), + signal: AbortSignal.timeout(10_000), + }); + if (!response.ok) throw new Error(`OTLP export → ${response.status}`); + await response.arrayBuffer().catch(() => {}); + this.stats.exports += 1; + this.stats.lastExportAt = Date.now(); + this._warnedOnce = false; + return true; + } catch (err) { + this.stats.failures += 1; + // Warn once per failure streak, then stay quiet — a down collector + // must not spam logs at every interval. + if (!this._warnedOnce) { + this._warnedOnce = true; + logger.warn({ endpoint, error: err.message }, '[Otel] metrics export failing (will keep retrying quietly)'); + } + return false; + } + } + + getStatus() { + return { enabled: !!_endpoint(), running: !!this.timer, intervalMs: EXPORT_INTERVAL_MS, ...this.stats }; + } +} + +let exporter = null; + +/** @returns {OtelExporter} singleton */ +function getOtelExporter() { + if (!exporter) exporter = new OtelExporter(); + return exporter; +} + +module.exports = { getOtelExporter, buildOtlpPayload }; diff --git a/src/routing/bandit.js b/src/routing/bandit.js index b6110ba..c2750e8 100644 --- a/src/routing/bandit.js +++ b/src/routing/bandit.js @@ -187,6 +187,45 @@ class LinUCBBandit { return best; } + /** + * Mean reward estimate for one arm in one context — θ·x with NO + * uncertainty bonus. This is the regression model r̂(x, a) the + * doubly-robust off-policy estimator consumes (routing/ope.js): the same + * ridge-regression state the live policy learns from, reused as the + * DR baseline instead of training a second model. + * + * Returns null when the arm has never been observed (identity-prior-only + * arms predict 0 by construction, which would bias DR toward "never + * tried" — better to let the estimator fall back to the IPS-only term). + * + * @param {string} tier + * @param {string} provider + * @param {string} model + * @param {number[]} context + * @returns {number|null} estimated reward in [0, 1], or null if unknown arm + */ + estimateReward(tier, provider, model, context) { + const key = this._armKey(tier, provider, model); + const arm = this.arms.get(key); + if (!arm || arm.count === 0) return null; + let ctx = context; + if (ctx.length !== this.dim) { + ctx = ctx.slice(0, this.dim); + while (ctx.length < this.dim) ctx.push(0); + } + let Ainv; + try { + Ainv = _inv(arm.A); + } catch { + return null; + } + const theta = _matVec(Ainv, arm.b); + const mean = _dot(theta, ctx); + // Rewards are trained in [0, 1]; clamp the linear extrapolation so a + // wild θ·x can't dominate the DR correction term. + return Math.max(0, Math.min(1, mean)); + } + /** * Update the chosen arm with the observed reward. * @param {string} tier diff --git a/src/routing/decide.js b/src/routing/decide.js new file mode 100644 index 0000000..b0182de --- /dev/null +++ b/src/routing/decide.js @@ -0,0 +1,149 @@ +/** + * Routing decision core — the harness-shaped, standalone pieces of the + * per-request model decision, extracted from routing/index.js so they can be + * consumed by more than one caller: + * + * - routing/index.js (the live request path, unchanged behavior) + * - the off-policy evaluator (WS4 follow-up), which must re-derive a + * candidate policy's π(a|x) on logged rows using the exact same context + * vector and candidate-eligibility rules the live policy used + * - a future remote decision endpoint, which needs decide-shaped logic + * that doesn't reach into request/response plumbing + * + * Deliberately excluded: session pins, the escalation ladder, kNN confidence + * branching, deadline/tenant overrides. Those are decision *inputs and + * post-passes* owned by the caller; this module owns only "given a context + * and a candidate set, pick one and report the probability of that pick." + * + * Nothing here catches errors — callers keep their existing + * degradation.record() wrapping so failure accounting stays where it was. + */ + +const { getBandit } = require('./bandit'); + +/** + * Task-type one-hot vocabulary. Order is load-bearing: it defines feature + * indices 6..11 of the context vector, and every persisted bandit arm was + * trained against this order. Do not reorder or insert — append only, and + * only together with a bandit-state migration. + */ +const TASK_TYPES = ['code_gen', 'summarization', 'reasoning', 'factoid', 'chat', 'other']; + +/** + * Build the 12-dim context vector the bandit (and any off-policy estimator + * replaying its decisions) scores against. + * + * Layout: [score, log-tokens, has-tools, streaming, risk, agentic, + * ...one-hot task type (6)]. + * + * @param {object} args + * @param {object} args.analysis — complexity analysis ({ score, breakdown }) + * @param {object} [args.payload] — request payload (for tools presence) + * @param {object} [args.options] — routing options (for streaming flag) + * @param {object} [args.risk] — risk analysis ({ level }) + * @param {object} [args.agenticResult] — agentic detection ({ isAgentic }) + * @returns {number[]} 12-dim feature vector + */ +function buildContextVector({ analysis, payload, options, risk, agenticResult }) { + const inferredTask = (analysis?.breakdown?.taskType?.reason || 'other').toLowerCase(); + const taskIdx = Math.max(0, TASK_TYPES.findIndex(t => inferredTask.includes(t))); + return [ + (analysis?.score || 0) / 100, + Math.log(Math.max(1, analysis?.breakdown?.tokenCount || 0) + 1) / 15, + ((payload?.tools?.length ?? 0) > 0) ? 1 : 0, + options?.streaming ? 1 : 0, + risk?.level === 'high' ? 1 : risk?.level === 'medium' ? 0.5 : 0, + agenticResult?.isAgentic ? 1 : 0, + ...TASK_TYPES.map((_, i) => i === taskIdx ? 1 : 0), + ]; +} + +/** + * Build the bandit's candidate set: the current selection plus the kNN + * alternative, if it differs AND is configured in some TIER_* entry. + * + * Tier-aware filter: the bandit may explore freely across the user's + * configured tiers (e.g. swap a SIMPLE request to the COMPLEX-tier model), + * but never pick a credentialed-but-untiered model (e.g. an Azure deployment + * present in .env for another purpose but referenced by no TIER_*). Tier + * routing stays the source of truth for eligibility. + * + * @param {{ provider: string, model: string }} current + * @param {{ provider: string, model: string }|null} alternative + * @returns {Array<{ provider: string, model: string }>} + */ +function buildCandidates(current, alternative) { + const candidates = [{ provider: current.provider, model: current.model }]; + if (alternative && alternative.model && alternative.model !== current.model) { + const configured = require('./model-tiers').getModelTierSelector().getAllConfiguredModels(); + const inConfig = configured.some( + m => m.provider === alternative.provider && m.model === alternative.model + ); + if (inConfig) { + candidates.push({ provider: alternative.provider, model: alternative.model }); + } + } + return candidates; +} + +/** + * The decision core: given a tier, a candidate set, and a context vector, + * have the bandit pick one and report the pick's propensity. + * + * Returns null when there's nothing to adjudicate (fewer than 2 candidates) + * or the bandit declined — callers keep their pre-existing selection. + * + * @param {string} tier + * @param {Array<{ provider, model }>} candidates + * @param {number[]} ctx — from buildContextVector + * @returns {null | { provider, model, ucb, explored, propensity, candidates, context }} + */ +function decide(tier, candidates, ctx) { + if (!candidates || candidates.length < 2) return null; + const picked = getBandit().pick(tier, candidates, ctx); + if (!picked) return null; + return { ...picked, candidates, context: ctx }; +} + +/** + * Stamp propensity/candidates/_banditContext onto a built decision (WS4.2). + * + * Collapse rule: the bandit's propensity only describes the served choice if + * the served (provider, model) is still one of the bandit's candidates. If a + * deterministic downstream override (deadline / tenant) swapped the served + * model out of that set — or the bandit never ran — collapse to + * propensity=1.0 with a single-entry candidate list, so off-policy + * estimators treat the row as a deterministic decision. + * + * _banditContext is underscored so it never leaks to response headers; the + * feedback path consumes it to call bandit.update(). + * + * @param {object} decision — mutated in place (matches prior inline behavior) + * @param {{ provider: string, model: string }} served + * @param {null | { propensity, candidates, context }} banditResult — from decide() + * @returns {object} the same decision, for chaining + */ +function stampPropensity(decision, served, banditResult) { + const banditPickedServed = banditResult?.candidates + && banditResult.candidates.some( + c => c.provider === served.provider && c.model === served.model + ); + if (banditPickedServed) { + decision.propensity = banditResult.propensity ?? 1.0; + decision.candidates = banditResult.candidates; + decision._banditContext = banditResult.context; + } else { + decision.propensity = 1.0; + decision.candidates = [{ provider: served.provider, model: served.model }]; + decision._banditContext = null; + } + return decision; +} + +module.exports = { + TASK_TYPES, + buildContextVector, + buildCandidates, + decide, + stampPropensity, +}; diff --git a/src/routing/deescalator.js b/src/routing/deescalator.js index 1a17579..ffa4199 100644 --- a/src/routing/deescalator.js +++ b/src/routing/deescalator.js @@ -103,6 +103,42 @@ function _clearCache() { _cache.clear(); } +/** + * Percentage-based demotion holdout (ROUTING-NOTES §4.10.4). + * + * Even when the evidence supports demotion, a configurable slice of sessions + * is deliberately excluded and served at the original tier. Those held-out + * rows are a continuously-running baseline: comparing their outcomes against + * demoted rows (method suffix '+deescalated' vs '+deescalation_holdout' in + * routing telemetry) proves the demotion rule is still net-positive over + * time, instead of trusting a one-time calibration. + * + * The bucket is a deterministic hash of the session key, so a session lands + * on the same side of the holdout on every turn — cohorts stay clean and a + * conversation never flip-flops tiers because of the holdout itself. + * + * LYNKR_DEESCALATION_HOLDOUT_PCT: 0–100, default 10. 0 disables the holdout. + * Read at call time (not module load) so tests and live re-config work. + * + * @param {string|null} sessionKey — session id/fingerprint; null → never held out + * @returns {boolean} true when this session must NOT be demoted (baseline cohort) + */ +function isHeldOut(sessionKey) { + if (!sessionKey) return false; + const raw = Number.parseInt(process.env.LYNKR_DEESCALATION_HOLDOUT_PCT, 10); + const pct = Number.isNaN(raw) ? 10 : Math.min(100, Math.max(0, raw)); + if (pct === 0) return false; + // FNV-1a over the session key → stable bucket in 0..99. + let hash = 0x811c9dc5; + const s = String(sessionKey); + for (let i = 0; i < s.length; i++) { + hash ^= s.charCodeAt(i); + hash = Math.imul(hash, 0x01000193); + } + const bucket = (hash >>> 0) % 100; + return bucket < pct; +} + /** * Shadow-mode policy. Wraps the live routing decision by delegating to * `determineProviderSmart` and then applying `suggestDemotion` on the result. @@ -143,6 +179,7 @@ async function shadowDeescalate(payload) { module.exports = { suggestDemotion, shadowDeescalate, + isHeldOut, TIER_ORDER, _clearCache, }; diff --git a/src/routing/index.js b/src/routing/index.js index a0d98ed..355f750 100644 --- a/src/routing/index.js +++ b/src/routing/index.js @@ -29,6 +29,7 @@ const { scoreIntent, intentScoreMode } = require('./intent-score'); // Phase 3-6 routing modules const { getKnnRouter } = require('./knn-router'); const { getBandit } = require('./bandit'); +const { buildCandidates, buildContextVector, decide, stampPropensity } = require('./decide'); const { getShadowPolicy, compareAndLog: shadowCompareAndLog } = require('./shadow-mode'); const { chooseFastest } = require('./deadline'); const { applyTenantOverrides } = require('./tenant-policy'); @@ -391,6 +392,11 @@ function _pinToDecision(pin, { reason, risk }) { async function determineProviderSmart(payload, options = {}) { const pinCheck = checkSessionPin(payload, options); + // Thread the session identity into fresh routing without mutating the + // caller's options object. Used by the de-escalation holdout so a session + // deterministically lands on one side of the holdout on every turn. + options = pinCheck.sessionId ? { ...options, _sessionId: pinCheck.sessionId } : options; + // Bypass (no session / forceProvider / feature-off) or no pin yet → // straight to fresh routing, then persist the outcome for the next turn. if (pinCheck.reason === 'bypass' || pinCheck.reason === 'no_pin') { @@ -680,6 +686,27 @@ function checkSessionPin(payload, options = {}) { const refreshOk = Array.isArray(payload?.tools) && payload.tools.length > 0; if (sessionAffinity.payloadHasToolHistory(payload)) { + // Stuck-loop detection: the pinned model re-issuing the same tool call + // (or the same text) over and over. Switching mid-exchange is forbidden + // (tool-call IDs aren't provider-portable), so the intervention is the + // same safe pattern as the embedded-text triggers below — serve the pin + // this turn but DROP it, so the next turn boundary re-routes fresh. + try { + const { detectStuckLoop } = require('./stuck-detector'); + const stuck = detectStuckLoop(payload); + if (stuck.stuck) { + sessionAffinity.removePin(sessionId); + logger.warn({ + sessionId, + trigger: stuck.reason, + repeats: stuck.repeats, + signature: stuck.signature, + pinnedTier: pin.tier, + pinnedModel: pin.model, + }, '[Routing] Stuck loop detected — pin dropped, next boundary re-routes fresh'); + return { serve: true, pin, reason: 'tool_history_pin_dropped', sessionId }; + } + } catch { /* never block the pin-serve path */ } // Text typed during a tool loop arrives merged with the pending // tool_result, where the pin serves unconditionally (id linkage forbids // switching mid-exchange). If that embedded text trips a trigger, drop @@ -1226,17 +1253,31 @@ async function _determineProviderSmartInner(payload, options = {}) { analysis, }); if (demoted && demoted !== tier) { - const demotedSelection = selector.selectModel(demoted, null); - logger.debug({ - from: `${tier}:${provider}:${selectedModel}`, - to: `${demoted}:${demotedSelection.provider}:${demotedSelection.model}`, - requestType, - }, '[Routing] De-escalation — demoting tier on evidence'); - demotedFrom = tier; - provider = demotedSelection.provider; - selectedModel = demotedSelection.model; - tier = demoted; - method = method + '+deescalated'; + // Percentage holdout (LYNKR_DEESCALATION_HOLDOUT_PCT, default 10): + // a deterministic slice of sessions is served at the original tier + // even though the evidence supports demotion. Their telemetry rows + // (method '+deescalation_holdout') are the running baseline that + // proves demoted sessions aren't quietly doing worse. + if (deescalator.isHeldOut(options._sessionId)) { + method = method + '+deescalation_holdout'; + logger.debug({ + tier, + wouldDemoteTo: demoted, + requestType, + }, '[Routing] De-escalation — evidence supports demotion, session held out as baseline'); + } else { + const demotedSelection = selector.selectModel(demoted, null); + logger.debug({ + from: `${tier}:${provider}:${selectedModel}`, + to: `${demoted}:${demotedSelection.provider}:${demotedSelection.model}`, + requestType, + }, '[Routing] De-escalation — demoting tier on evidence'); + demotedFrom = tier; + provider = demotedSelection.provider; + selectedModel = demotedSelection.model; + tier = demoted; + method = method + '+deescalated'; + } } } catch (err) { degradation.record('tier_select', err); @@ -1467,65 +1508,29 @@ async function _determineProviderSmartInner(payload, options = {}) { // WS4.2 — capture propensity + candidates + context so the outcome row can // support off-policy evaluation. banditContext is stashed on the decision // (underscored → won't leak through headers, see WS4.2 verification). - let banditPropensity = null; - let banditCandidates = null; - let banditContext = null; + // + // The candidate-building / context-vector / pick logic lives in decide.js + // so the off-policy evaluator can replay it against logged rows. + let banditResult = null; if (config.routing?.banditEnabled !== false && knnResult && knnResult.model) { try { - // Build candidates: current selection and kNN alternative if different. - // - // Tier-aware filter: only treat the kNN suggestion as a real candidate - // if it matches a (provider, model) combo configured in ANY TIER_* - // entry. The bandit is allowed to explore freely across the user's - // configured tiers (e.g. swap a SIMPLE request to the COMPLEX-tier - // model), but is forbidden from picking a credentialed-but-untiered - // model (e.g. an Azure OpenAI deployment whose endpoint is set in .env - // for some other use, but not referenced by any TIER_*). This keeps - // tier routing as the source of truth for what's eligible while - // preserving cross-tier bandit exploration. - const allCandidates = [{ provider, model: selectedModel }]; - if (knnResult.model !== selectedModel) { - const configured = require('./model-tiers').getModelTierSelector().getAllConfiguredModels(); - const inConfig = configured.some( - m => m.provider === knnResult.provider && m.model === knnResult.model - ); - if (inConfig) { - allCandidates.push({ provider: knnResult.provider, model: knnResult.model }); - } - } - - if (allCandidates.length > 1) { - const bandit = getBandit(); - const TASK_TYPES = ['code_gen', 'summarization', 'reasoning', 'factoid', 'chat', 'other']; - const inferredTask = (analysis.breakdown?.taskType?.reason || 'other').toLowerCase(); - const taskIdx = Math.max(0, TASK_TYPES.findIndex(t => inferredTask.includes(t))); - const ctx = [ - (analysis.score || 0) / 100, - Math.log(Math.max(1, analysis.breakdown?.tokenCount || 0) + 1) / 15, - ((payload?.tools?.length ?? 0) > 0) ? 1 : 0, - options.streaming ? 1 : 0, - risk?.level === 'high' ? 1 : risk?.level === 'medium' ? 0.5 : 0, - agenticResult?.isAgentic ? 1 : 0, - ...TASK_TYPES.map((_, i) => i === taskIdx ? 1 : 0), - ]; - const picked = bandit.pick(tier, allCandidates, ctx); - if (picked) { - banditCandidates = allCandidates; - banditPropensity = picked.propensity ?? null; - banditContext = ctx; - if (picked.model !== selectedModel) { - logger.debug({ - from: `${provider}:${selectedModel}`, - to: `${picked.provider}:${picked.model}`, - ucb: picked.ucb?.toFixed(4), - explored: picked.explored, - propensity: picked.propensity, - }, '[Routing] Bandit override'); - provider = picked.provider; - selectedModel = picked.model; - method = method + (picked.explored ? '+bandit_explore' : '+bandit'); - } - } + const allCandidates = buildCandidates( + { provider, model: selectedModel }, + { provider: knnResult.provider, model: knnResult.model }, + ); + const ctx = buildContextVector({ analysis, payload, options, risk, agenticResult }); + banditResult = decide(tier, allCandidates, ctx); + if (banditResult && banditResult.model !== selectedModel) { + logger.debug({ + from: `${provider}:${selectedModel}`, + to: `${banditResult.provider}:${banditResult.model}`, + ucb: banditResult.ucb?.toFixed(4), + explored: banditResult.explored, + propensity: banditResult.propensity, + }, '[Routing] Bandit override'); + provider = banditResult.provider; + selectedModel = banditResult.model; + method = method + (banditResult.explored ? '+bandit_explore' : '+bandit'); } } catch (err) { degradation.record('bandit', err); @@ -1603,24 +1608,13 @@ async function _determineProviderSmartInner(payload, options = {}) { }); // WS4.2 — propensity/candidates for off-policy evaluation from telemetry. - // Bandit picks populate both. If a deterministic downstream override - // (deadline / tenant) then swapped the served model out of the bandit's - // candidate set, the bandit's propensity no longer describes the served - // choice — collapse to propensity=1.0 with a single candidate. Otherwise - // deterministic branches (bandit didn't run at all) always collapse. - // _banditContext is underscored so it never leaks to response headers; - // WS5 will consume it in the feedback path to call bandit.update(). - const banditPickedServed = banditCandidates - && banditCandidates.some(c => c.provider === provider && c.model === selectedModel); - if (banditPickedServed) { - decision.propensity = banditPropensity ?? 1.0; - decision.candidates = banditCandidates; - decision._banditContext = banditContext; - } else { - decision.propensity = 1.0; - decision.candidates = [{ provider, model: selectedModel }]; - decision._banditContext = null; - } + // Collapse rule lives in decide.js (stampPropensity): if a deterministic + // downstream override (deadline / tenant) swapped the served model out of + // the bandit's candidate set — or the bandit never ran — the row collapses + // to propensity=1.0 with a single candidate. _banditContext is underscored + // so it never leaks to response headers; the feedback path consumes it to + // call bandit.update(). + stampPropensity(decision, { provider, model: selectedModel }, banditResult); // WS5.5 — attach the query embedding + raw query text so the feedback // path can turn conclusive outcomes into new kNN exemplars without diff --git a/src/routing/ope.js b/src/routing/ope.js new file mode 100644 index 0000000..0be6368 --- /dev/null +++ b/src/routing/ope.js @@ -0,0 +1,255 @@ +/** + * Off-policy evaluation (ROUTING-NOTES §3 + §4.10.3). + * + * Estimates how a CANDIDATE routing policy would have performed, using only + * logged decisions made by the LIVE policy — no live traffic, no A/B test. + * This consumes exactly what WS4.2 has been logging on every telemetry row + * since it shipped: `propensity` (the live policy's probability of the + * logged choice), `candidates` (the choice set), `quality_score` (reward), + * and — as of the same change that added this module — `context` (the + * bandit's 12-dim feature vector, needed by the doubly-robust term). + * + * Four estimators, in increasing order of sophistication: + * + * IPS importance sampling: reweight logged rewards by + * π(a|x) / p. Unbiased but high-variance when propensities are + * small. + * SNIPS self-normalized IPS: divide by the sum of weights instead of n. + * Slightly biased, far lower variance (Swaminathan & Joachims). + * DR doubly robust (Dudík/Langford/Li 2011): a regression baseline + * r̂(x, a) — LinUCB's own per-arm ridge model, reused — plus the + * importance-weighted residual. Unbiased if EITHER the propensities + * OR the regression model is right. + * WDR weighted DR: DR with SNIPS-style self-normalized weights on the + * correction term. The production target named by real systems' + * own code ("the IPS/DR/WDR estimators"). + * + * A candidate policy is a plain function: + * policyFn({ tier, context, candidates }) → + * { probs: Map } — action probabilities summing + * to 1 over the row's candidate set (deterministic policies return + * probability 1 on one candidate). + * + * Rewards are quality_score rescaled to [0, 1] (same scale the bandit and + * its r̂ train on). Rows are only usable when they carry propensity, + * candidates (≥1), and quality_score; the DR term additionally needs + * context and falls back to the pure IPS term on rows without it. + * + * @module routing/ope + */ + +const { getBandit } = require('./bandit'); + +/** Canonical key for a candidate. */ +function candKey(c) { + return `${c.provider}:${c.model}`; +} + +// Propensity floor: a logged propensity below this is clamped, bounding any +// single row's importance weight (the same reason workweave's Monte-Carlo +// propensity floors at 1/trials — one row must not dominate the estimate). +const PROPENSITY_FLOOR = 1e-3; + +function _parseRow(row) { + if (row == null) return null; + const propensity = typeof row.propensity === 'number' ? row.propensity : null; + const quality = typeof row.quality_score === 'number' ? row.quality_score : null; + if (propensity == null || quality == null) return null; + + let candidates = row.candidates; + if (typeof candidates === 'string') { + try { + candidates = JSON.parse(candidates); + } catch { + return null; + } + } + if (!Array.isArray(candidates) || candidates.length === 0) return null; + + let context = row.context ?? null; + if (typeof context === 'string') { + try { + context = JSON.parse(context); + } catch { + context = null; + } + } + if (!Array.isArray(context)) context = null; + + return { + tier: row.tier ?? null, + served: { provider: row.provider, model: row.model ?? null }, + reward: Math.max(0, Math.min(1, quality / 100)), + propensity: Math.max(PROPENSITY_FLOOR, Math.min(1, propensity)), + candidates, + context, + }; +} + +/** + * Evaluate a candidate policy against logged telemetry rows. + * + * @param {Array} rows — telemetry rows (raw DB rows or equivalents) + * @param {Function} policyFn — ({ tier, context, candidates }) → { probs: Map } + * @param {object} [deps] + * @param {object} [deps.bandit] — override the reward model (tests) + * @returns {{ + * n: number, usable: number, drRows: number, + * ips: number|null, snips: number|null, dr: number|null, wdr: number|null, + * effectiveSampleSize: number|null, + * loggedMeanReward: number|null, + * }} + */ +function evaluatePolicy(rows, policyFn, deps = {}) { + const bandit = deps.bandit || getBandit(); + + let usable = 0; + let drRows = 0; + let ipsSum = 0; + let weightSum = 0; + let weightSqSum = 0; + let drSum = 0; + let wdrBaselineSum = 0; + let wdrCorrectionSum = 0; + let loggedRewardSum = 0; + + for (const raw of rows) { + const row = _parseRow(raw); + if (!row) continue; + usable += 1; + loggedRewardSum += row.reward; + + let probs; + try { + const out = policyFn({ tier: row.tier, context: row.context, candidates: row.candidates }); + probs = out?.probs; + } catch { + probs = null; + } + if (!probs) continue; + + // π(a_logged | x): the candidate policy's probability of the action the + // live policy actually served. + const piLogged = probs.get(candKey(row.served)) ?? 0; + const w = piLogged / row.propensity; + + ipsSum += w * row.reward; + weightSum += w; + weightSqSum += w * w; + + // DR terms: baseline = Σ_a π(a|x)·r̂(x,a); correction = w·(r − r̂(x,a_logged)). + // Rows without a context (bandit didn't run) or without a usable r̂ fall + // back to r̂ = 0, which reduces the row's DR contribution to pure IPS. + let baseline = 0; + let rhatLogged = 0; + if (row.context && row.tier) { + drRows += 1; + for (const c of row.candidates) { + const p = probs.get(candKey(c)) ?? 0; + if (p === 0) continue; + const rhat = bandit.estimateReward(row.tier, c.provider, c.model, row.context); + if (rhat != null) baseline += p * rhat; + } + const rhatServed = bandit.estimateReward(row.tier, row.served.provider, row.served.model, row.context); + if (rhatServed != null) rhatLogged = rhatServed; + } + drSum += baseline + w * (row.reward - rhatLogged); + wdrBaselineSum += baseline; + wdrCorrectionSum += w * (row.reward - rhatLogged); + } + + if (usable === 0) { + return { + n: rows.length, usable: 0, drRows: 0, + ips: null, snips: null, dr: null, wdr: null, + effectiveSampleSize: null, loggedMeanReward: null, + }; + } + + // Effective sample size — Kish's approximation. A small ESS relative to + // `usable` means a few high-weight rows dominate: treat the estimate with + // suspicion regardless of its value. + const ess = weightSqSum > 0 ? (weightSum * weightSum) / weightSqSum : 0; + + return { + n: rows.length, + usable, + drRows, + ips: ipsSum / usable, + snips: weightSum > 0 ? ipsSum / weightSum : null, + dr: drSum / usable, + // WDR: baseline averaged per-row (deterministic, no weights) plus the + // self-normalized correction — variance control on exactly the term + // that needs it. + wdr: weightSum > 0 + ? wdrBaselineSum / usable + wdrCorrectionSum / weightSum + : wdrBaselineSum / usable, + effectiveSampleSize: ess, + loggedMeanReward: loggedRewardSum / usable, + }; +} + +// --------------------------------------------------------------------------- +// Reference policies — useful baselines to evaluate out of the box. +// --------------------------------------------------------------------------- + +/** Always pick the first candidate (the tier-config/heuristic selection). */ +function firstCandidatePolicy({ candidates }) { + const probs = new Map(); + candidates.forEach((c, i) => probs.set(candKey(c), i === 0 ? 1 : 0)); + return { probs }; +} + +/** Always pick the last candidate (the kNN suggestion when present). */ +function lastCandidatePolicy({ candidates }) { + const probs = new Map(); + const lastIdx = candidates.length - 1; + candidates.forEach((c, i) => probs.set(candKey(c), i === lastIdx ? 1 : 0)); + return { probs }; +} + +/** Uniform-random over the candidate set. */ +function uniformPolicy({ candidates }) { + const probs = new Map(); + const p = 1 / candidates.length; + candidates.forEach((c) => probs.set(candKey(c), p)); + return { probs }; +} + +/** + * The current LinUCB policy replayed greedily (no exploration, mean+bonus + * argmax at TODAY'S learned weights). Comparing this against the logged + * rewards answers: "has the bandit's learning actually converged on + * something better than what it served while learning?" + */ +function currentBanditGreedyPolicy({ tier, context, candidates }, deps = {}) { + const bandit = deps.bandit || getBandit(); + const probs = new Map(); + if (!context || !tier || candidates.length < 2) { + return firstCandidatePolicy({ candidates }); + } + let best = null; + let bestVal = -Infinity; + for (const c of candidates) { + const est = bandit.estimateReward(tier, c.provider, c.model, context); + const val = est ?? -1; // unknown arms lose to any known arm + if (val > bestVal) { + bestVal = val; + best = c; + } + } + candidates.forEach((c) => probs.set(candKey(c), c === best ? 1 : 0)); + return { probs }; +} + +module.exports = { + evaluatePolicy, + candKey, + PROPENSITY_FLOOR, + policies: { + firstCandidate: firstCandidatePolicy, + lastCandidate: lastCandidatePolicy, + uniform: uniformPolicy, + currentBanditGreedy: currentBanditGreedyPolicy, + }, +}; diff --git a/src/routing/stuck-detector.js b/src/routing/stuck-detector.js new file mode 100644 index 0000000..1f8cf7e --- /dev/null +++ b/src/routing/stuck-detector.js @@ -0,0 +1,127 @@ +/** + * Stuck-loop / no-progress detection (ROUTING-NOTES §4.10.5). + * + * The escalation ladder reacts to score drift, context overflow, risk + * keywords, and vision needs — but nothing watches for "the pinned model is + * thrashing": re-issuing the same tool call with the same input over and + * over, or repeating the same assistant text verbatim. A cheap-tier model + * stuck in that state burns turns indefinitely, because every frame of the + * loop is a mid-tool-exchange pin serve (which is unconditional — tool-call + * IDs aren't portable across providers, so switching mid-exchange would 400). + * + * The safe intervention is the one the pin path already uses for embedded + * text triggers: don't switch this turn — DROP THE PIN, so the next turn + * boundary re-routes fresh (and full routing, seeing the whole struggling + * conversation, escalates on its own signals). + * + * Detection is deliberately narrow to keep false positives near zero: + * - tool repetition: the last K assistant tool_use blocks are the SAME + * tool with the SAME input, K >= LYNKR_STUCK_TOOL_REPEATS (default 3). + * Agents legitimately re-run a tool (poll, retry-once); three identical + * consecutive calls is a loop. + * - text repetition: the last K assistant text blocks are identical after + * whitespace normalization, K >= LYNKR_STUCK_TEXT_REPEATS (default 3). + * + * Only the tail of the conversation is scanned (SCAN_WINDOW messages), so + * the check is O(1)-ish per request regardless of conversation length. + * + * @module routing/stuck-detector + */ + +const SCAN_WINDOW = 20; + +function _toolRepeats() { + const n = Number.parseInt(process.env.LYNKR_STUCK_TOOL_REPEATS, 10); + return Number.isNaN(n) ? 3 : Math.max(2, n); +} + +function _textRepeats() { + const n = Number.parseInt(process.env.LYNKR_STUCK_TEXT_REPEATS, 10); + return Number.isNaN(n) ? 3 : Math.max(2, n); +} + +function _enabled() { + return process.env.LYNKR_STUCK_DETECTOR_ENABLED !== 'false'; +} + +/** + * Extract, oldest→newest, the assistant tool_use signatures and text blocks + * from the tail of the conversation. + */ +function _assistantTail(messages) { + const tail = messages.slice(-SCAN_WINDOW); + const toolSigs = []; + const texts = []; + for (const msg of tail) { + if (msg?.role !== 'assistant') continue; + const content = Array.isArray(msg.content) ? msg.content : []; + for (const block of content) { + if (block?.type === 'tool_use') { + let inputSig; + try { + inputSig = JSON.stringify(block.input ?? null); + } catch { + inputSig = String(block.input); + } + toolSigs.push(`${block.name}::${inputSig}`); + } else if (block?.type === 'text' && typeof block.text === 'string') { + const normalized = block.text.replace(/\s+/g, ' ').trim(); + if (normalized.length > 0) texts.push(normalized); + } + } + // String-content assistant messages count as text blocks too. + if (typeof msg.content === 'string') { + const normalized = msg.content.replace(/\s+/g, ' ').trim(); + if (normalized.length > 0) texts.push(normalized); + } + } + return { toolSigs, texts }; +} + +function _trailingRun(items) { + if (items.length === 0) return 0; + const last = items[items.length - 1]; + let run = 0; + for (let i = items.length - 1; i >= 0 && items[i] === last; i--) run++; + return run; +} + +/** + * Detect a stuck loop in the conversation tail. + * + * @param {object} payload — request payload with .messages + * @returns {{ stuck: boolean, reason?: 'tool_repetition'|'text_repetition', + * repeats?: number, signature?: string }} + */ +function detectStuckLoop(payload) { + if (!_enabled()) return { stuck: false }; + const messages = payload?.messages; + if (!Array.isArray(messages) || messages.length < 4) return { stuck: false }; + + const { toolSigs, texts } = _assistantTail(messages); + + const toolRun = _trailingRun(toolSigs); + if (toolRun >= _toolRepeats()) { + return { + stuck: true, + reason: 'tool_repetition', + repeats: toolRun, + // Truncated for logging — the full input may be huge or sensitive. + signature: toolSigs[toolSigs.length - 1].slice(0, 120), + }; + } + + const textRun = _trailingRun(texts); + if (textRun >= _textRepeats()) { + return { + stuck: true, + reason: 'text_repetition', + repeats: textRun, + signature: texts[texts.length - 1].slice(0, 120), + }; + } + + return { stuck: false }; +} + +module.exports = { detectStuckLoop }; diff --git a/src/routing/telemetry.js b/src/routing/telemetry.js index 81caca2..08bda28 100644 --- a/src/routing/telemetry.js +++ b/src/routing/telemetry.js @@ -175,6 +175,12 @@ function init() { // providers that report none) — "not measured" is distinct from 0. ["cache_read_tokens", "INTEGER"], ["cache_creation_tokens", "INTEGER"], + // Off-policy evaluation — the bandit's context vector (12-dim JSON + // array) for the row's decision. Required by the doubly-robust + // estimator's regression term r̂(x, a); IPS/SNIPS work without it. + // NULL on rows where the bandit didn't run (deterministic decisions) + // or recorded before this column existed. + ["context", "TEXT"], ]; for (const [col, type] of additiveCols) { if (!existingCols.has(col)) { @@ -237,7 +243,7 @@ function record(data) { retry_count, circuit_breaker_state, quality_score, tokens_per_second, cost_efficiency, request_text, response_text, base_tier, escalation_source, propensity, candidates, pinned, switch_reason, - cache_decision, cache_read_tokens, cache_creation_tokens + cache_decision, cache_read_tokens, cache_creation_tokens, context ) VALUES ( @request_id, @session_id, @timestamp, @complexity_score, @tier, @agentic_type, @tool_count, @input_tokens, @message_count, @request_type, @@ -246,7 +252,7 @@ function record(data) { @retry_count, @circuit_breaker_state, @quality_score, @tokens_per_second, @cost_efficiency, @request_text, @response_text, @base_tier, @escalation_source, @propensity, @candidates, @pinned, @switch_reason, - @cache_decision, @cache_read_tokens, @cache_creation_tokens + @cache_decision, @cache_read_tokens, @cache_creation_tokens, @context )` ); if (!insert) return; @@ -299,6 +305,9 @@ function record(data) { : JSON.stringify(data.cache_decision)), cache_read_tokens: data.cache_read_tokens ?? null, cache_creation_tokens: data.cache_creation_tokens ?? null, + context: data.context == null + ? null + : (typeof data.context === "string" ? data.context : JSON.stringify(data.context)), }); } catch (err) { logger.debug({ err: err.message }, "Telemetry record failed"); diff --git a/src/server.js b/src/server.js index 458472d..2c73f3b 100644 --- a/src/server.js +++ b/src/server.js @@ -109,7 +109,8 @@ function createApp() { app.get("/metrics/circuit-breakers", (req, res) => { const registry = getCircuitBreakerRegistry(); - res.json(registry.getAll()); + const { getHealthProber } = require("./clients/health-probe"); + res.json({ breakers: registry.getAll(), healthProbe: getHealthProber().getStatus() }); }); app.get("/metrics/load-shedding", (req, res) => { @@ -128,13 +129,23 @@ function createApp() { app.get("/metrics/semantic-cache", (req, res) => { const { getSemanticCache, isSemanticCacheEnabled } = require("./cache/semantic"); + const { getEmbeddingStatus } = require("./cache/embeddings"); if (!isSemanticCacheEnabled()) { return res.json({ enabled: false, message: "Semantic cache not enabled" }); } const cache = getSemanticCache(); - res.json({ enabled: true, ...cache.getStats() }); + // embeddings.providerAvailable === false means matches are currently + // served from the non-semantic hash fallback — cache "hits" during a + // degraded window are approximate, not semantic. + res.json({ enabled: true, embeddings: getEmbeddingStatus(), ...cache.getStats() }); }); + // MCP broker — exposes configured MCP servers to other clients over HTTP. + // Off by default; requires LYNKR_MCP_BROKER_ENABLED + a bearer token. + // Mounted BEFORE the main router so /v1/mcp/* never falls through to the + // OpenAI-compat surface. + app.use(require('./api/mcp-broker')); + app.use(router); app.use('/dashboard', require('./dashboard/router')); @@ -185,6 +196,29 @@ async function start() { const app = createApp(); + // Synthetic circuit-breaker health probing — recovers open circuits with a + // cheap background probe instead of letting the next live user request pay + // for testing a dead provider. Zero-cost while all breakers are closed. + try { + const { getHealthProber } = require("./clients/health-probe"); + const prober = getHealthProber(); + prober.start(); + getShutdownManager().onShutdown(() => prober.stop()); + } catch (err) { + logger.warn({ err: err.message }, "Health prober failed to start, circuit recovery falls back to live-request probing"); + } + + // OTel GenAI metrics export — no-op unless OTEL_EXPORTER_OTLP_ENDPOINT + // (or LYNKR_OTEL_ENDPOINT) is set. Zero-dependency OTLP/HTTP push. + try { + const { getOtelExporter } = require("./observability/otel"); + const otel = getOtelExporter(); + otel.start(); + getShutdownManager().onShutdown(() => otel.stop()); + } catch (err) { + logger.warn({ err: err.message }, "OTel exporter failed to start"); + } + // Wait for Ollama if it's the configured provider or referenced in tier config const provider = config.modelProvider?.type?.toLowerCase(); if (provider === "ollama" || config.tiersReferenceOllama()) { diff --git a/test/decide.test.js b/test/decide.test.js new file mode 100644 index 0000000..3a9e0f6 --- /dev/null +++ b/test/decide.test.js @@ -0,0 +1,110 @@ +/** + * Contract tests for src/routing/decide.js — the extracted decision core. + * + * These pin the module's own behavior (context-vector layout, candidate + * eligibility, propensity collapse rule) independently of routing/index.js, + * because the off-policy evaluator will import these functions directly to + * replay logged decisions — a silent contract drift here would corrupt + * counterfactual estimates without failing any routing test. + */ + +const test = require('node:test'); +const assert = require('node:assert/strict'); + +process.env.DATABRICKS_API_KEY = process.env.DATABRICKS_API_KEY || 'test-key'; +process.env.DATABRICKS_API_BASE = process.env.DATABRICKS_API_BASE || 'http://test.com'; +process.env.LOG_FILE_ENABLED = 'false'; + +const { TASK_TYPES, buildContextVector, decide, stampPropensity } = require('../src/routing/decide'); + +test('context vector is 12-dim: 6 scalar features + 6-way task one-hot', () => { + const ctx = buildContextVector({ + analysis: { score: 50, breakdown: { tokenCount: 100, taskType: { reason: 'code_gen' } } }, + payload: { tools: [{}] }, + options: { streaming: true }, + risk: { level: 'medium' }, + agenticResult: { isAgentic: true }, + }); + assert.equal(ctx.length, 6 + TASK_TYPES.length); + assert.equal(ctx[0], 0.5); // score/100 + assert.equal(ctx[1], Math.log(101) / 15); // log(tokens+1)/15 + assert.equal(ctx[2], 1); // has tools + assert.equal(ctx[3], 1); // streaming + assert.equal(ctx[4], 0.5); // medium risk + assert.equal(ctx[5], 1); // agentic + assert.deepEqual(ctx.slice(6), [1, 0, 0, 0, 0, 0]); // code_gen one-hot +}); + +test('context vector defaults: empty inputs produce the "other" task one-hot', () => { + const ctx = buildContextVector({ analysis: {} }); + assert.equal(ctx[0], 0); + assert.equal(ctx[4], 0); // no risk info → 0 + // 'other' is index 5 in TASK_TYPES + assert.deepEqual(ctx.slice(6), [0, 0, 0, 0, 0, 1]); +}); + +test('decide returns null when there is nothing to adjudicate', () => { + const ctx = buildContextVector({ analysis: { score: 10 } }); + assert.equal(decide('SIMPLE', [], ctx), null); + assert.equal(decide('SIMPLE', [{ provider: 'ollama', model: 'qwen' }], ctx), null); + assert.equal(decide('SIMPLE', null, ctx), null); +}); + +test('decide returns a pick carrying propensity, candidates, and context', () => { + const ctx = buildContextVector({ analysis: { score: 10 } }); + const candidates = [ + { provider: 'ollama', model: 'qwen' }, + { provider: 'databricks', model: 'claude-sonnet' }, + ]; + const result = decide('SIMPLE', candidates, ctx); + assert.ok(result, 'expected a pick with 2 candidates'); + assert.ok(candidates.some(c => c.model === result.model), 'pick must be one of the candidates'); + assert.ok(result.propensity > 0 && result.propensity <= 1, `propensity in (0,1], got ${result.propensity}`); + assert.deepEqual(result.candidates, candidates); + assert.deepEqual(result.context, ctx); +}); + +test('stampPropensity: served model in bandit candidate set keeps bandit propensity + context', () => { + const decision = {}; + const banditResult = { + propensity: 0.9625, + candidates: [ + { provider: 'ollama', model: 'qwen' }, + { provider: 'databricks', model: 'claude-sonnet' }, + ], + context: [0.1, 0.2], + }; + stampPropensity(decision, { provider: 'databricks', model: 'claude-sonnet' }, banditResult); + assert.equal(decision.propensity, 0.9625); + assert.equal(decision.candidates.length, 2); + assert.deepEqual(decision._banditContext, [0.1, 0.2]); +}); + +test('stampPropensity: downstream override swapping the served model collapses to 1.0', () => { + const decision = {}; + const banditResult = { + propensity: 0.9625, + candidates: [{ provider: 'ollama', model: 'qwen' }], + context: [0.1], + }; + // Tenant/deadline override served something the bandit never considered. + stampPropensity(decision, { provider: 'azure', model: 'gpt-4o' }, banditResult); + assert.equal(decision.propensity, 1.0); + assert.deepEqual(decision.candidates, [{ provider: 'azure', model: 'gpt-4o' }]); + assert.equal(decision._banditContext, null); +}); + +test('stampPropensity: bandit never ran (null result) collapses to 1.0', () => { + const decision = {}; + stampPropensity(decision, { provider: 'ollama', model: 'qwen' }, null); + assert.equal(decision.propensity, 1.0); + assert.deepEqual(decision.candidates, [{ provider: 'ollama', model: 'qwen' }]); + assert.equal(decision._banditContext, null); +}); + +test('TASK_TYPES order is frozen — bandit arms were trained against these indices', () => { + // Reordering or inserting entries silently corrupts every persisted arm's + // learned weights (feature indices 6..11 shift). Append-only, with a + // bandit-state migration. This test makes that drift loud. + assert.deepEqual(TASK_TYPES, ['code_gen', 'summarization', 'reasoning', 'factoid', 'chat', 'other']); +}); diff --git a/test/deescalator.test.js b/test/deescalator.test.js index cbcf499..8f2bbf9 100644 --- a/test/deescalator.test.js +++ b/test/deescalator.test.js @@ -159,3 +159,71 @@ test('cache invalidated after TTL', () => { }); assert.equal(calls, 2); }); + +// --- Percentage holdout (ROUTING-NOTES §4.10.4) --------------------------- + +const { isHeldOut } = require('../src/routing/deescalator'); + +test('holdout: null session key is never held out', () => { + process.env.LYNKR_DEESCALATION_HOLDOUT_PCT = '50'; + assert.equal(isHeldOut(null), false); + assert.equal(isHeldOut(undefined), false); + delete process.env.LYNKR_DEESCALATION_HOLDOUT_PCT; +}); + +test('holdout: 0 pct disables the holdout entirely', () => { + process.env.LYNKR_DEESCALATION_HOLDOUT_PCT = '0'; + for (let i = 0; i < 50; i++) { + assert.equal(isHeldOut(`session-${i}`), false); + } + delete process.env.LYNKR_DEESCALATION_HOLDOUT_PCT; +}); + +test('holdout: 100 pct holds every session out', () => { + process.env.LYNKR_DEESCALATION_HOLDOUT_PCT = '100'; + for (let i = 0; i < 50; i++) { + assert.equal(isHeldOut(`session-${i}`), true); + } + delete process.env.LYNKR_DEESCALATION_HOLDOUT_PCT; +}); + +test('holdout: deterministic — same session key always lands on the same side', () => { + process.env.LYNKR_DEESCALATION_HOLDOUT_PCT = '30'; + for (let i = 0; i < 20; i++) { + const key = `fp-abc${i}`; + const first = isHeldOut(key); + for (let j = 0; j < 5; j++) { + assert.equal(isHeldOut(key), first, `session ${key} flip-flopped`); + } + } + delete process.env.LYNKR_DEESCALATION_HOLDOUT_PCT; +}); + +test('holdout: observed rate roughly tracks the configured percentage', () => { + process.env.LYNKR_DEESCALATION_HOLDOUT_PCT = '20'; + let held = 0; + const N = 2000; + for (let i = 0; i < N; i++) { + if (isHeldOut(`fp-${i}-${i * 7919}`)) held++; + } + const rate = held / N; + assert.ok(rate > 0.15 && rate < 0.25, `expected ~0.20 holdout rate, got ${rate}`); + delete process.env.LYNKR_DEESCALATION_HOLDOUT_PCT; +}); + +test('holdout: default (env unset) is 10 pct, and invalid values clamp sanely', () => { + delete process.env.LYNKR_DEESCALATION_HOLDOUT_PCT; + let held = 0; + const N = 2000; + for (let i = 0; i < N; i++) { + if (isHeldOut(`fp-${i}-${i * 104729}`)) held++; + } + const rate = held / N; + assert.ok(rate > 0.06 && rate < 0.14, `expected ~0.10 default holdout rate, got ${rate}`); + + process.env.LYNKR_DEESCALATION_HOLDOUT_PCT = '250'; + assert.equal(isHeldOut('any-session'), true, '>100 clamps to 100'); + process.env.LYNKR_DEESCALATION_HOLDOUT_PCT = '-5'; + assert.equal(isHeldOut('any-session'), false, 'negative clamps to 0'); + delete process.env.LYNKR_DEESCALATION_HOLDOUT_PCT; +}); diff --git a/test/embeddings-degradation.test.js b/test/embeddings-degradation.test.js new file mode 100644 index 0000000..f9bffee --- /dev/null +++ b/test/embeddings-degradation.test.js @@ -0,0 +1,101 @@ +/** + * Embedding-provider degradation behavior (ROUTING-NOTES §4.10.1). + * + * The hash fallback is non-semantic — while it's active, semantic cache and + * kNN matching are effectively disabled. Previously a single provider + * failure latched the fallback permanently (until process restart) and + * logged only at debug level. These tests pin the fixed behavior: + * + * 1. a provider failure degrades to the hash fallback (fail-soft kept) + * 2. the degradation is NOT a permanent latch — after the retry cooldown, + * the provider is re-attempted and recovery is automatic + * 3. within the cooldown, the dead provider is not re-dialed per request + * 4. getEmbeddingStatus() reports the degraded state for /metrics + */ + +const test = require('node:test'); +const assert = require('node:assert/strict'); +const http = require('http'); + +process.env.DATABRICKS_API_KEY = process.env.DATABRICKS_API_KEY || 'test-key'; +process.env.DATABRICKS_API_BASE = process.env.DATABRICKS_API_BASE || 'http://test.com'; +process.env.LOG_FILE_ENABLED = 'false'; +// Short cooldown so the re-attempt path is testable without sleeping 60s. +process.env.LYNKR_EMBEDDINGS_RETRY_COOLDOWN_MS = '150'; + +// A controllable fake Ollama /api/embeddings endpoint. +let failMode = false; +let providerHits = 0; +const fakeOllama = http.createServer((req, res) => { + providerHits += 1; + if (failMode) { + res.statusCode = 500; + res.end('{"error":"down"}'); + return; + } + res.setHeader('Content-Type', 'application/json'); + res.end(JSON.stringify({ embedding: new Array(768).fill(0.5) })); +}); + +let embeddings; + +test.before(async () => { + await new Promise((resolve) => fakeOllama.listen(0, '127.0.0.1', resolve)); + const { port } = fakeOllama.address(); + process.env.OLLAMA_EMBEDDINGS_ENDPOINT = `http://127.0.0.1:${port}/api/embeddings`; + // Fresh config + module so the endpoint override is picked up. + delete require.cache[require.resolve('../src/config')]; + delete require.cache[require.resolve('../src/cache/embeddings')]; + embeddings = require('../src/cache/embeddings'); +}); + +test.after(() => fakeOllama.close()); + +test.beforeEach(() => { + embeddings.resetEmbeddingProvider(); + failMode = false; + providerHits = 0; +}); + +test('healthy provider: real 768-dim embedding, status available', async () => { + const vec = await embeddings.generateEmbedding('hello world'); + assert.equal(vec.length, 768); + const status = embeddings.getEmbeddingStatus(); + assert.equal(status.providerAvailable, true); + assert.equal(status.fallbackCount, 0); +}); + +test('provider failure degrades to hash fallback and reports degraded status', async () => { + failMode = true; + const vec = await embeddings.generateEmbedding('hello world'); + assert.equal(vec.length, 384, 'hash fallback is 384-dim'); + const status = embeddings.getEmbeddingStatus(); + assert.equal(status.providerAvailable, false); + assert.ok(status.degradedSince > 0, 'degradedSince timestamp set'); + assert.ok(status.fallbackCount >= 1); + assert.match(status.lastProviderError, /500/); +}); + +test('within the cooldown, the dead provider is not re-dialed per request', async () => { + failMode = true; + await embeddings.generateEmbedding('first — triggers degradation'); + const hitsAfterFirst = providerHits; + await embeddings.generateEmbedding('second — must be served from fallback without dialing'); + await embeddings.generateEmbedding('third — same'); + assert.equal(providerHits, hitsAfterFirst, 'no additional provider dials inside the cooldown window'); +}); + +test('degradation is not a permanent latch: provider recovery is automatic after cooldown', async () => { + failMode = true; + const degraded = await embeddings.generateEmbedding('while down'); + assert.equal(degraded.length, 384); + + failMode = false; + await new Promise((r) => setTimeout(r, 200)); // > LYNKR_EMBEDDINGS_RETRY_COOLDOWN_MS + + const recovered = await embeddings.generateEmbedding('after recovery'); + assert.equal(recovered.length, 768, 'real embeddings resume after the provider comes back'); + const status = embeddings.getEmbeddingStatus(); + assert.equal(status.providerAvailable, true); + assert.equal(status.degradedSince, null); +}); diff --git a/test/health-probe.test.js b/test/health-probe.test.js new file mode 100644 index 0000000..d401c6f --- /dev/null +++ b/test/health-probe.test.js @@ -0,0 +1,108 @@ +/** + * Synthetic circuit-breaker health probing (ROUTING-NOTES §1 gap audit). + * + * Pins the core contract: an OPEN circuit recovers via a background + * synthetic probe — no live user request has to pay for testing a dead + * provider — and healthy (CLOSED) breakers are never probed at all. + */ + +const test = require('node:test'); +const assert = require('node:assert/strict'); + +process.env.DATABRICKS_API_KEY = process.env.DATABRICKS_API_KEY || 'test-key'; +process.env.DATABRICKS_API_BASE = process.env.DATABRICKS_API_BASE || 'http://test.com'; +process.env.LOG_FILE_ENABLED = 'false'; + +const { getCircuitBreakerRegistry } = require('../src/clients/circuit-breaker'); +const { getHealthProber, registerHealthProbe } = require('../src/clients/health-probe'); + +const FAILURE_THRESHOLD = 3; +const HALF_OPEN_AFTER_MS = 60; + +async function forceOpen(breaker) { + for (let i = 0; i < FAILURE_THRESHOLD; i++) { + await breaker.execute(() => Promise.reject(new Error('provider down'))).catch(() => {}); + } + assert.equal(breaker.state, 'OPEN', 'breaker should be open after threshold failures'); +} + +test('open circuit recovers via synthetic probe once the provider is healthy again', async () => { + const registry = getCircuitBreakerRegistry(); + const breaker = registry.get('probe-test-recovers', { + failureThreshold: FAILURE_THRESHOLD, + timeout: HALF_OPEN_AFTER_MS, + }); + await forceOpen(breaker); + + let providerHealthy = false; + let probeDials = 0; + registerHealthProbe('probe-test-recovers', async () => { + probeDials += 1; + if (!providerHealthy) throw new Error('still down'); + }); + + const prober = getHealthProber(); + + // Sweep while the provider is still down and the circuit is inside its + // halfOpenAfter window: execute() rejects without dialing the probe. + await prober.sweep(); + assert.equal(breaker.state, 'OPEN'); + + // Provider comes back; after halfOpenAfter elapses, a sweep's probe runs + // through the half-open circuit and closes it — no live request involved. + providerHealthy = true; + await new Promise((r) => setTimeout(r, HALF_OPEN_AFTER_MS + 20)); + await prober.sweep(); + assert.equal(breaker.state, 'CLOSED', 'synthetic probe should close the recovered circuit'); + assert.ok(probeDials >= 1, 'probe function actually dialed the provider'); +}); + +test('probe failure while provider is down re-opens (keeps open) the circuit', async () => { + const registry = getCircuitBreakerRegistry(); + const breaker = registry.get('probe-test-stays-open', { + failureThreshold: FAILURE_THRESHOLD, + timeout: HALF_OPEN_AFTER_MS, + }); + await forceOpen(breaker); + + registerHealthProbe('probe-test-stays-open', async () => { + throw new Error('still down'); + }); + + await new Promise((r) => setTimeout(r, HALF_OPEN_AFTER_MS + 20)); + await getHealthProber().sweep(); + assert.equal(breaker.state, 'OPEN', 'failed probe must leave the circuit open'); +}); + +test('healthy (CLOSED) breakers are never probed — zero cost when everything is up', async () => { + const registry = getCircuitBreakerRegistry(); + const breaker = registry.get('probe-test-healthy', { + failureThreshold: FAILURE_THRESHOLD, + timeout: HALF_OPEN_AFTER_MS, + }); + assert.equal(breaker.state, 'CLOSED'); + + let probeDials = 0; + registerHealthProbe('probe-test-healthy', async () => { + probeDials += 1; + }); + + await getHealthProber().sweep(); + assert.equal(probeDials, 0, 'closed breakers must not be probed'); +}); + +test('providers without a registered breaker are skipped', async () => { + let probeDials = 0; + registerHealthProbe('probe-test-no-breaker-exists', async () => { + probeDials += 1; + }); + await getHealthProber().sweep(); + assert.equal(probeDials, 0); +}); + +test('getStatus reports registered providers and sweep counters', async () => { + const status = getHealthProber().getStatus(); + assert.ok(status.sweeps >= 1, 'sweeps counted'); + assert.ok(status.registeredProviders.includes('probe-test-recovers')); + assert.equal(typeof status.intervalMs, 'number'); +}); diff --git a/test/hierarchical-budget.test.js b/test/hierarchical-budget.test.js index 79a1d48..4f4b0d1 100644 --- a/test/hierarchical-budget.test.js +++ b/test/hierarchical-budget.test.js @@ -50,3 +50,35 @@ test('missing context level is ignored', () => { const r = b.check({ team: null, customer: null }, 100); assert.equal(r.ok, true); }); + +// --- Pre-flight cost estimation (ROUTING-NOTES §1 gap: flat $0.01 stub) ---- + +const { estimateRequestCost } = require('../src/api/middleware/budget-enforcer'); + +test('estimateRequestCost floors at the old $0.01 nominal gate', () => { + // Empty/unpriceable payloads keep the legacy behavior exactly. + assert.ok(estimateRequestCost({}) >= 0.01); + assert.ok(estimateRequestCost(null) >= 0.01); +}); + +test('estimateRequestCost grows with payload size', () => { + const small = estimateRequestCost({ + messages: [{ role: 'user', content: 'hi' }], + max_tokens: 100, + }); + const large = estimateRequestCost({ + system: 'x'.repeat(40_000), + messages: [{ role: 'user', content: 'y'.repeat(400_000) }], + max_tokens: 8000, + }); + // With any non-zero blended pricing the large payload must cost more; with + // zero pricing both floor at 0.01 — either way large >= small always holds. + assert.ok(large >= small, `large (${large}) should be >= small (${small})`); +}); + +test('estimateRequestCost respects caller max_tokens over the output floor', () => { + const base = { messages: [{ role: 'user', content: 'hello world' }] }; + const smallOut = estimateRequestCost({ ...base, max_tokens: 1 }); + const bigOut = estimateRequestCost({ ...base, max_tokens: 32_000 }); + assert.ok(bigOut >= smallOut); +}); diff --git a/test/mcp-broker.test.js b/test/mcp-broker.test.js new file mode 100644 index 0000000..043f736 --- /dev/null +++ b/test/mcp-broker.test.js @@ -0,0 +1,173 @@ +/** + * MCP broker (ROUTING-NOTES Track B — Lynkr was MCP-client-only). + * + * Security contract is the load-bearing part: + * - disabled by default → 404 (the surface doesn't exist) + * - enabled without a token → 503 fail-closed (never unauthenticated) + * - wrong/missing bearer → 401 + * Function contract: aggregated namespaced tool list, per-server errors + * reported rather than hidden, calls proxied to the underlying client. + */ + +const test = require('node:test'); +const assert = require('node:assert/strict'); +const http = require('http'); +const express = require('express'); + +process.env.DATABRICKS_API_KEY = process.env.DATABRICKS_API_KEY || 'test-key'; +process.env.DATABRICKS_API_BASE = process.env.DATABRICKS_API_BASE || 'http://test.com'; +process.env.LOG_FILE_ENABLED = 'false'; + +// Mock the MCP module before the broker requires it. +const mcpPath = require.resolve('../src/mcp'); +const fakeClients = { + alpha: { + request: async (method, params) => { + if (method === 'tools/list') { + return { tools: [{ name: 'read_file', description: 'Read a file', inputSchema: { type: 'object' } }] }; + } + if (method === 'tools/call') return { content: [{ type: 'text', text: `alpha ran ${params.name}` }] }; + throw new Error(`unexpected method ${method}`); + }, + }, + beta: { + request: async (method) => { + if (method === 'tools/list') throw new Error('beta is down'); + throw new Error('beta is down'); + }, + }, +}; +require.cache[mcpPath] = { + id: mcpPath, + filename: mcpPath, + loaded: true, + exports: { + listServers: () => [ + { id: 'alpha', description: 'files' }, + { id: 'beta', description: 'flaky' }, + ], + getServer: (id) => (['alpha', 'beta'].includes(id) ? { id } : null), + ensureClient: async (id) => { + const c = fakeClients[id]; + if (!c) throw new Error(`no client ${id}`); + return c; + }, + }, +}; + +const brokerRouter = require('../src/api/mcp-broker'); + +let server; +let port; + +test.before(async () => { + const app = express(); + app.use(express.json()); + app.use(brokerRouter); + server = await new Promise((resolve) => { + const s = app.listen(0, '127.0.0.1', () => resolve(s)); + }); + port = server.address().port; +}); + +test.after(() => { + server.close(); + delete require.cache[mcpPath]; +}); + +test.beforeEach(() => { + delete process.env.LYNKR_MCP_BROKER_ENABLED; + delete process.env.LYNKR_MCP_BROKER_TOKEN; + delete process.env.LYNKR_MCP_BROKER_SERVERS; +}); + +function call(pathname, { method = 'GET', token, body } = {}) { + return new Promise((resolve, reject) => { + const data = body ? JSON.stringify(body) : null; + const req = http.request({ + host: '127.0.0.1', + port, + path: pathname, + method, + headers: { + ...(token ? { Authorization: `Bearer ${token}` } : {}), + ...(data ? { 'Content-Type': 'application/json', 'Content-Length': Buffer.byteLength(data) } : {}), + }, + }, (res) => { + let chunks = ''; + res.on('data', (c) => (chunks += c)); + res.on('end', () => resolve({ status: res.statusCode, body: JSON.parse(chunks || '{}') })); + }); + req.on('error', reject); + if (data) req.write(data); + req.end(); + }); +} + +test('disabled by default: the surface does not exist (404)', async () => { + const res = await call('/v1/mcp/tools'); + assert.equal(res.status, 404); +}); + +test('enabled without a token fails CLOSED (503), never serves', async () => { + process.env.LYNKR_MCP_BROKER_ENABLED = 'true'; + const res = await call('/v1/mcp/tools'); + assert.equal(res.status, 503); + assert.equal(res.body.error.type, 'broker_misconfigured'); +}); + +test('wrong or missing bearer → 401', async () => { + process.env.LYNKR_MCP_BROKER_ENABLED = 'true'; + process.env.LYNKR_MCP_BROKER_TOKEN = 'secret-token'; + assert.equal((await call('/v1/mcp/tools')).status, 401); + assert.equal((await call('/v1/mcp/tools', { token: 'wrong' })).status, 401); +}); + +test('tool list aggregates across servers, namespaced, with per-server errors reported', async () => { + process.env.LYNKR_MCP_BROKER_ENABLED = 'true'; + process.env.LYNKR_MCP_BROKER_TOKEN = 'secret-token'; + const res = await call('/v1/mcp/tools', { token: 'secret-token' }); + assert.equal(res.status, 200); + assert.equal(res.body.tools.length, 1); + assert.equal(res.body.tools[0].name, 'alpha:read_file'); + assert.equal(res.body.tools[0].server, 'alpha'); + // beta's failure is visible, not silently dropped (§4.3). + assert.match(res.body.server_errors.beta, /beta is down/); +}); + +test('tools/call proxies to the underlying client', async () => { + process.env.LYNKR_MCP_BROKER_ENABLED = 'true'; + process.env.LYNKR_MCP_BROKER_TOKEN = 'secret-token'; + const res = await call('/v1/mcp/tools/call', { + method: 'POST', + token: 'secret-token', + body: { server: 'alpha', tool: 'read_file', arguments: { path: '/x' } }, + }); + assert.equal(res.status, 200); + assert.equal(res.body.result.content[0].text, 'alpha ran read_file'); +}); + +test('unknown server → 404; server allowlist enforced → 403', async () => { + process.env.LYNKR_MCP_BROKER_ENABLED = 'true'; + process.env.LYNKR_MCP_BROKER_TOKEN = 'secret-token'; + const unknown = await call('/v1/mcp/tools/call', { + method: 'POST', token: 'secret-token', body: { server: 'nope', tool: 'x' }, + }); + assert.equal(unknown.status, 404); + + process.env.LYNKR_MCP_BROKER_SERVERS = 'beta'; + const blocked = await call('/v1/mcp/tools/call', { + method: 'POST', token: 'secret-token', body: { server: 'alpha', tool: 'read_file' }, + }); + assert.equal(blocked.status, 403); +}); + +test('call failure surfaces as an error status, not a fake success', async () => { + process.env.LYNKR_MCP_BROKER_ENABLED = 'true'; + process.env.LYNKR_MCP_BROKER_TOKEN = 'secret-token'; + const res = await call('/v1/mcp/tools/call', { + method: 'POST', token: 'secret-token', body: { server: 'beta', tool: 'anything' }, + }); + assert.equal(res.status, 502); + assert.equal(res.body.error.type, 'tool_call_failed'); +}); diff --git a/test/onnx-embedder.test.js b/test/onnx-embedder.test.js new file mode 100644 index 0000000..bade812 --- /dev/null +++ b/test/onnx-embedder.test.js @@ -0,0 +1,108 @@ +/** + * ONNX embedder wiring (ROUTING-NOTES §4.10.2). + * + * Unit-level tests only: provider-branch selection, availability detection, + * and fallthrough when the optionalDependency is missing. Real inference + * (model download + embed) is deliberately NOT exercised here — it pulls + * ~30MB from the HuggingFace hub and takes ~1 min cold. That path is + * verified manually / in integration, not in the unit suite. + */ + +const test = require('node:test'); +const assert = require('node:assert/strict'); + +process.env.DATABRICKS_API_KEY = process.env.DATABRICKS_API_KEY || 'test-key'; +process.env.DATABRICKS_API_BASE = process.env.DATABRICKS_API_BASE || 'http://test.com'; +process.env.LOG_FILE_ENABLED = 'false'; + +function freshEmbeddings() { + delete require.cache[require.resolve('../src/cache/embeddings')]; + return require('../src/cache/embeddings'); +} + +function mockOnnxEmbedder({ available, vector }) { + const modulePath = require.resolve('../src/cache/onnx-embedder'); + delete require.cache[modulePath]; + require.cache[modulePath] = { + id: modulePath, + filename: modulePath, + loaded: true, + exports: { + isOnnxAvailable: () => available, + generateOnnxEmbedding: async () => vector, + _resetPipeline: () => {}, + MODEL_ID: 'mock-model', + }, + }; +} + +function restoreOnnxEmbedder() { + const modulePath = require.resolve('../src/cache/onnx-embedder'); + delete require.cache[modulePath]; +} + +test.afterEach(() => { + delete process.env.LYNKR_EMBEDDINGS_PROVIDER; + restoreOnnxEmbedder(); +}); + +test('isOnnxAvailable reflects the installed optionalDependency', () => { + const { isOnnxAvailable } = require('../src/cache/onnx-embedder'); + // In this repo the dep is installed; the check must agree with require.resolve. + let resolvable = true; + try { require.resolve('@huggingface/transformers'); } catch { resolvable = false; } + assert.equal(isOnnxAvailable(), resolvable); +}); + +test('LYNKR_EMBEDDINGS_PROVIDER=onnx routes embedding through the in-process embedder', async () => { + process.env.LYNKR_EMBEDDINGS_PROVIDER = 'onnx'; + const fake = new Array(768).fill(0.25); + mockOnnxEmbedder({ available: true, vector: fake }); + const embeddings = freshEmbeddings(); + embeddings.resetEmbeddingProvider(); + + const vec = await embeddings.generateEmbedding('hello'); + assert.equal(vec.length, 768); + assert.equal(vec[0], 0.25); + const status = embeddings.getEmbeddingStatus(); + assert.equal(status.providerAvailable, true, 'onnx path marks the provider healthy'); +}); + +test('onnx requested but dependency missing falls through to the network provider chain', async () => { + process.env.LYNKR_EMBEDDINGS_PROVIDER = 'onnx'; + mockOnnxEmbedder({ available: false, vector: null }); + const embeddings = freshEmbeddings(); + embeddings.resetEmbeddingProvider(); + + // No Ollama/llamacpp endpoints reachable in unit tests — the chain ends at + // the hash fallback (384-dim). The point: no crash, and the onnx branch + // was skipped rather than erroring. + const vec = await embeddings.generateEmbedding('hello'); + assert.ok(vec.length === 384 || vec.length === 768, + `expected hash fallback (384) or a real provider vector (768), got ${vec.length}`); +}); + +test('onnx load failure degrades loudly via the shared degradation machinery', async () => { + process.env.LYNKR_EMBEDDINGS_PROVIDER = 'onnx'; + const modulePath = require.resolve('../src/cache/onnx-embedder'); + delete require.cache[modulePath]; + require.cache[modulePath] = { + id: modulePath, + filename: modulePath, + loaded: true, + exports: { + isOnnxAvailable: () => true, + generateOnnxEmbedding: async () => { throw new Error('model load failed'); }, + _resetPipeline: () => {}, + MODEL_ID: 'mock-model', + }, + }; + const embeddings = freshEmbeddings(); + embeddings.resetEmbeddingProvider(); + + const vec = await embeddings.generateEmbedding('hello'); + assert.equal(vec.length, 384, 'hash fallback served'); + const status = embeddings.getEmbeddingStatus(); + assert.equal(status.providerAvailable, false, 'degraded state is recorded, not silent'); + assert.match(status.lastProviderError, /model load failed/); +}); diff --git a/test/ope.test.js b/test/ope.test.js new file mode 100644 index 0000000..a5923f5 --- /dev/null +++ b/test/ope.test.js @@ -0,0 +1,153 @@ +/** + * Off-policy evaluation estimators (ROUTING-NOTES §3 + §4.10.3). + * + * Synthetic ground-truth experiments: construct logged data from a known + * behavior policy over arms with known true rewards, then verify each + * estimator recovers the true value of a target policy. The doubly-robust + * property itself is tested directly: corrupt the logged propensities and + * confirm DR (with a correct reward model) still recovers truth while IPS + * does not. + */ + +const test = require('node:test'); +const assert = require('node:assert/strict'); + +process.env.DATABRICKS_API_KEY = process.env.DATABRICKS_API_KEY || 'test-key'; +process.env.DATABRICKS_API_BASE = process.env.DATABRICKS_API_BASE || 'http://test.com'; +process.env.LOG_FILE_ENABLED = 'false'; + +const { evaluatePolicy, policies } = require('../src/routing/ope'); + +// Two arms with known true rewards (quality points, 0-100 scale in logs). +const ARM_A = { provider: 'ollama', model: 'qwen' }; // true reward 80 +const ARM_B = { provider: 'azure', model: 'gpt-4o' }; // true reward 40 +const TRUE_REWARD = { 'ollama:qwen': 80, 'azure:gpt-4o': 40 }; +const CANDIDATES = [ARM_A, ARM_B]; +const CTX = [0.5, 0.2, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0]; + +/** + * Deterministic logged dataset: the behavior policy served A on exactly + * pA-fraction of rows and B on the rest, propensities logged accordingly, + * rewards exactly at the true means (noise-free so estimates are exact). + */ +function makeLogs({ n, pA, loggedPropensityA = pA, withContext = true }) { + const rows = []; + const nA = Math.round(n * pA); + for (let i = 0; i < n; i++) { + const servedA = i < nA; + const served = servedA ? ARM_A : ARM_B; + rows.push({ + tier: 'MEDIUM', + provider: served.provider, + model: served.model, + quality_score: TRUE_REWARD[`${served.provider}:${served.model}`], + propensity: servedA ? loggedPropensityA : 1 - loggedPropensityA, + candidates: JSON.stringify(CANDIDATES), + context: withContext ? JSON.stringify(CTX) : null, + }); + } + return rows; +} + +/** A reward model that knows the truth exactly (context-independent). */ +const perfectBandit = { + estimateReward: (tier, provider, model) => + (TRUE_REWARD[`${provider}:${model}`] ?? null) === null + ? null + : TRUE_REWARD[`${provider}:${model}`] / 100, +}; + +/** A reward model that knows nothing. */ +const ignorantBandit = { estimateReward: () => null }; + +const alwaysA = ({ candidates }) => { + const probs = new Map(); + candidates.forEach((c) => probs.set(`${c.provider}:${c.model}`, c.model === ARM_A.model ? 1 : 0)); + return { probs }; +}; + +test('IPS/SNIPS recover the target policy value from correctly-logged propensities', () => { + const rows = makeLogs({ n: 1000, pA: 0.7 }); + const r = evaluatePolicy(rows, alwaysA, { bandit: ignorantBandit }); + // True value of always-A = 0.8. Deterministic construction → exact. + assert.ok(Math.abs(r.ips - 0.8) < 1e-9, `IPS ${r.ips} != 0.8`); + assert.ok(Math.abs(r.snips - 0.8) < 1e-9, `SNIPS ${r.snips} != 0.8`); + assert.equal(r.usable, 1000); + // Logged policy served a 0.7/0.3 mix → logged mean = 0.7·0.8 + 0.3·0.4 = 0.68. + assert.ok(Math.abs(r.loggedMeanReward - 0.68) < 1e-9); +}); + +test('DR and WDR agree with IPS when both propensities and model are correct', () => { + const rows = makeLogs({ n: 1000, pA: 0.7 }); + const r = evaluatePolicy(rows, alwaysA, { bandit: perfectBandit }); + assert.ok(Math.abs(r.dr - 0.8) < 1e-9, `DR ${r.dr} != 0.8`); + assert.ok(Math.abs(r.wdr - 0.8) < 1e-9, `WDR ${r.wdr} != 0.8`); + assert.equal(r.drRows, 1000); +}); + +test('THE doubly-robust property: corrupted propensities bias IPS but not DR', () => { + // Behavior policy truly served A 70% of the time, but the logs LIE and say + // propensity was 0.5 for every A row (and 0.5 for B). + const rows = makeLogs({ n: 1000, pA: 0.7, loggedPropensityA: 0.5 }); + + const biased = evaluatePolicy(rows, alwaysA, { bandit: ignorantBandit }); + // IPS with wrong propensities: 0.7n rows · (1/0.5)·0.8 / n = 1.12 ≠ 0.8. + assert.ok(Math.abs(biased.ips - 1.12) < 1e-9, `expected biased IPS 1.12, got ${biased.ips}`); + + const robust = evaluatePolicy(rows, alwaysA, { bandit: perfectBandit }); + // With a correct r̂, the residual (r − r̂) is 0 on every row, so the + // corrupted weights multiply zero: DR = Σ π·r̂ = 0.8 exactly. + assert.ok(Math.abs(robust.dr - 0.8) < 1e-9, `DR ${robust.dr} != 0.8 under corrupted propensities`); + assert.ok(Math.abs(robust.wdr - 0.8) < 1e-9, `WDR ${robust.wdr} != 0.8 under corrupted propensities`); +}); + +test('rows without context fall back to the IPS term inside DR (no crash, coverage reported)', () => { + const rows = makeLogs({ n: 500, pA: 0.7, withContext: false }); + const r = evaluatePolicy(rows, alwaysA, { bandit: perfectBandit }); + assert.equal(r.drRows, 0, 'no context → no DR regression coverage'); + assert.ok(Math.abs(r.dr - 0.8) < 1e-9, 'DR degenerates to IPS and still recovers truth'); +}); + +test('unusable rows (missing propensity/quality/candidates) are skipped, not fatal', () => { + const rows = [ + ...makeLogs({ n: 100, pA: 0.7 }), + { provider: 'x', model: 'y' }, // no propensity/quality + { propensity: 0.5, quality_score: 50, candidates: 'not-json' }, // bad JSON + null, + ]; + const r = evaluatePolicy(rows, alwaysA, { bandit: ignorantBandit }); + assert.equal(r.usable, 100); + assert.equal(r.n, 103); +}); + +test('effective sample size equals usable rows when weights are uniform', () => { + // Uniform target over 2 candidates: w = 0.5/p, and with pA=0.5 every row + // has identical weight → ESS = usable. + const rows = makeLogs({ n: 400, pA: 0.5 }); + const r = evaluatePolicy(rows, policies.uniform, { bandit: ignorantBandit }); + assert.ok(Math.abs(r.effectiveSampleSize - 400) < 1e-6, `ESS ${r.effectiveSampleSize} != 400`); +}); + +test('propensity floor bounds a single row\'s importance weight', () => { + const rows = [{ + tier: 'MEDIUM', + provider: ARM_A.provider, + model: ARM_A.model, + quality_score: 80, + propensity: 1e-9, // absurd logged propensity + candidates: JSON.stringify(CANDIDATES), + context: JSON.stringify(CTX), + }]; + const r = evaluatePolicy(rows, alwaysA, { bandit: ignorantBandit }); + // Weight is clamped to 1/PROPENSITY_FLOOR = 1000, not 1e9. + assert.ok(r.ips <= 0.8 * 1000 + 1e-9, `weight not floored: IPS ${r.ips}`); +}); + +test('reference policies produce valid distributions over the candidate set', () => { + for (const [name, fn] of Object.entries(policies)) { + const { probs } = fn({ tier: 'MEDIUM', context: CTX, candidates: CANDIDATES }); + let sum = 0; + for (const c of CANDIDATES) sum += probs.get(`${c.provider}:${c.model}`) ?? 0; + assert.ok(Math.abs(sum - 1) < 1e-9, `${name} probabilities sum to ${sum}, not 1`); + } +}); diff --git a/test/otel-export.test.js b/test/otel-export.test.js new file mode 100644 index 0000000..84d624e --- /dev/null +++ b/test/otel-export.test.js @@ -0,0 +1,87 @@ +/** + * OTel GenAI metrics export (ROUTING-NOTES §1 gap audit). + * + * Zero-dependency OTLP/HTTP push + gen_ai.* semconv naming on the + * Prometheus surface. Tests: payload shape, live POST to a fake collector, + * disabled-by-default behavior, and the Prometheus aliases. + */ + +const test = require('node:test'); +const assert = require('node:assert/strict'); +const http = require('http'); + +process.env.DATABRICKS_API_KEY = process.env.DATABRICKS_API_KEY || 'test-key'; +process.env.DATABRICKS_API_BASE = process.env.DATABRICKS_API_BASE || 'http://test.com'; +process.env.LOG_FILE_ENABLED = 'false'; + +const { buildOtlpPayload, getOtelExporter } = require('../src/observability/otel'); +const { getMetricsCollector } = require('../src/observability/metrics'); + +const SAMPLE_METRICS = { + tokens_input_total: 1000, + tokens_output_total: 250, + cost_usd_total: 1.25, + requests_total: 40, + requests_errors_total: 2, + latency_ms: { median: 120, p95: 800, p99: 2000 }, +}; + +test('OTLP payload follows the ExportMetricsServiceRequest shape with gen_ai semconv names', () => { + const payload = buildOtlpPayload(SAMPLE_METRICS, { startTimeMs: 1000, nowMs: 61_000 }); + const scope = payload.resourceMetrics[0].scopeMetrics[0]; + const names = scope.metrics.map((m) => m.name); + assert.ok(names.includes('gen_ai.client.token.usage')); + assert.ok(names.includes('gen_ai.client.operation.duration')); + + const tokenUsage = scope.metrics.find((m) => m.name === 'gen_ai.client.token.usage'); + assert.equal(tokenUsage.sum.isMonotonic, true); + assert.equal(tokenUsage.sum.aggregationTemporality, 2, 'CUMULATIVE'); + const byType = Object.fromEntries( + tokenUsage.sum.dataPoints.map((p) => [p.attributes[0].value.stringValue, p.asDouble]) + ); + assert.equal(byType.input, 1000); + assert.equal(byType.output, 250); + + const resourceAttrs = Object.fromEntries( + payload.resourceMetrics[0].resource.attributes.map((a) => [a.key, a.value.stringValue]) + ); + assert.equal(resourceAttrs['service.name'], 'lynkr'); +}); + +test('exporter POSTs to /v1/metrics and reports success', async (t) => { + let received = null; + const collector = http.createServer((req, res) => { + let body = ''; + req.on('data', (c) => (body += c)); + req.on('end', () => { + received = { url: req.url, body: JSON.parse(body) }; + res.statusCode = 200; + res.end('{}'); + }); + }); + await new Promise((r) => collector.listen(0, '127.0.0.1', r)); + t.after(() => collector.close()); + + process.env.OTEL_EXPORTER_OTLP_ENDPOINT = `http://127.0.0.1:${collector.address().port}`; + t.after(() => delete process.env.OTEL_EXPORTER_OTLP_ENDPOINT); + + const ok = await getOtelExporter().exportOnce(); + assert.equal(ok, true); + assert.equal(received.url, '/v1/metrics'); + assert.ok(received.body.resourceMetrics, 'OTLP body delivered'); +}); + +test('export is a no-op when no endpoint is configured', async () => { + delete process.env.OTEL_EXPORTER_OTLP_ENDPOINT; + delete process.env.LYNKR_OTEL_ENDPOINT; + const ok = await getOtelExporter().exportOnce(); + assert.equal(ok, false); + assert.equal(getOtelExporter().getStatus().enabled, false); +}); + +test('Prometheus surface carries the gen_ai_* semconv aliases', () => { + const text = getMetricsCollector().toPrometheus(); + assert.match(text, /gen_ai_client_token_usage_total\{gen_ai_token_type="input"\}/); + assert.match(text, /gen_ai_client_token_usage_total\{gen_ai_token_type="output"\}/); + assert.match(text, /gen_ai_client_operation_duration_ms\{quantile="0.95"\}/); +}); diff --git a/test/stuck-detector.test.js b/test/stuck-detector.test.js new file mode 100644 index 0000000..d992a12 --- /dev/null +++ b/test/stuck-detector.test.js @@ -0,0 +1,141 @@ +/** + * Stuck-loop detection (ROUTING-NOTES §4.10.5). + * + * Narrow-by-design detector: three identical consecutive assistant tool + * calls (same name + same input) or three identical assistant text blocks. + * Legitimate agent behavior — different inputs to the same tool, polls with + * changing arguments, retry-once — must NOT trip it. + */ + +const test = require('node:test'); +const assert = require('node:assert/strict'); + +process.env.DATABRICKS_API_KEY = process.env.DATABRICKS_API_KEY || 'test-key'; +process.env.DATABRICKS_API_BASE = process.env.DATABRICKS_API_BASE || 'http://test.com'; +process.env.LOG_FILE_ENABLED = 'false'; + +const { detectStuckLoop } = require('../src/routing/stuck-detector'); + +function assistantToolCall(name, input) { + return { + role: 'assistant', + content: [{ type: 'tool_use', id: `tu_${Math.random().toString(36).slice(2)}`, name, input }], + }; +} + +function toolResult(text) { + return { role: 'user', content: [{ type: 'tool_result', tool_use_id: 'tu_x', content: text }] }; +} + +function assistantText(text) { + return { role: 'assistant', content: [{ type: 'text', text }] }; +} + +function conversation(...turns) { + return { messages: [{ role: 'user', content: 'do the thing' }, ...turns] }; +} + +test('three identical consecutive tool calls trip tool_repetition', () => { + const payload = conversation( + assistantToolCall('Read', { file: '/a.js' }), + toolResult('contents'), + assistantToolCall('Read', { file: '/a.js' }), + toolResult('contents'), + assistantToolCall('Read', { file: '/a.js' }), + toolResult('contents'), + ); + const result = detectStuckLoop(payload); + assert.equal(result.stuck, true); + assert.equal(result.reason, 'tool_repetition'); + assert.equal(result.repeats, 3); +}); + +test('same tool with DIFFERENT inputs is normal agent behavior — no trip', () => { + const payload = conversation( + assistantToolCall('Read', { file: '/a.js' }), + toolResult('a'), + assistantToolCall('Read', { file: '/b.js' }), + toolResult('b'), + assistantToolCall('Read', { file: '/c.js' }), + toolResult('c'), + ); + assert.equal(detectStuckLoop(payload).stuck, false); +}); + +test('two identical calls (retry-once) do not trip', () => { + const payload = conversation( + assistantToolCall('Bash', { cmd: 'npm test' }), + toolResult('flaky failure'), + assistantToolCall('Bash', { cmd: 'npm test' }), + toolResult('pass'), + ); + assert.equal(detectStuckLoop(payload).stuck, false); +}); + +test('a loop broken by a different call does not trip (run must be trailing)', () => { + const payload = conversation( + assistantToolCall('Read', { file: '/a.js' }), + toolResult('x'), + assistantToolCall('Read', { file: '/a.js' }), + toolResult('x'), + assistantToolCall('Write', { file: '/a.js', content: 'fixed' }), + toolResult('ok'), + ); + assert.equal(detectStuckLoop(payload).stuck, false); +}); + +test('three identical assistant text blocks trip text_repetition', () => { + const payload = conversation( + assistantText('I cannot complete this task.'), + { role: 'user', content: 'try again' }, + assistantText('I cannot complete this task.'), // whitespace normalized + { role: 'user', content: 'please try again' }, + assistantText('I cannot complete this task.'), + ); + const result = detectStuckLoop(payload); + assert.equal(result.stuck, true); + assert.equal(result.reason, 'text_repetition'); +}); + +test('varied assistant text does not trip', () => { + const payload = conversation( + assistantText('Reading the file now.'), + { role: 'user', content: 'ok' }, + assistantText('Found the bug on line 42.'), + { role: 'user', content: 'fix it' }, + assistantText('Fixed and tests pass.'), + ); + assert.equal(detectStuckLoop(payload).stuck, false); +}); + +test('short conversations never trip', () => { + assert.equal(detectStuckLoop({ messages: [{ role: 'user', content: 'hi' }] }).stuck, false); + assert.equal(detectStuckLoop({ messages: [] }).stuck, false); + assert.equal(detectStuckLoop({}).stuck, false); +}); + +test('detector can be disabled via env', () => { + process.env.LYNKR_STUCK_DETECTOR_ENABLED = 'false'; + const payload = conversation( + assistantToolCall('Read', { file: '/a.js' }), + toolResult('x'), + assistantToolCall('Read', { file: '/a.js' }), + toolResult('x'), + assistantToolCall('Read', { file: '/a.js' }), + toolResult('x'), + ); + assert.equal(detectStuckLoop(payload).stuck, false); + delete process.env.LYNKR_STUCK_DETECTOR_ENABLED; +}); + +test('repeat threshold is env-tunable', () => { + process.env.LYNKR_STUCK_TOOL_REPEATS = '2'; + const payload = conversation( + assistantToolCall('Bash', { cmd: 'npm test' }), + toolResult('fail'), + assistantToolCall('Bash', { cmd: 'npm test' }), + toolResult('fail'), + ); + assert.equal(detectStuckLoop(payload).stuck, true); + delete process.env.LYNKR_STUCK_TOOL_REPEATS; +}); diff --git a/test/token-rate-limit.test.js b/test/token-rate-limit.test.js new file mode 100644 index 0000000..8c67d2b --- /dev/null +++ b/test/token-rate-limit.test.js @@ -0,0 +1,90 @@ +/** + * Token-aware (TPM) rate limiting (ROUTING-NOTES §1 gap audit). + * + * Off unless LYNKR_TPM_LIMIT is set. Estimate → true-up pattern: pre-flight + * gates on the window's recorded actual consumption plus this request's + * estimate; actual usage lands post-response via recordTokenUsage. + */ + +const test = require('node:test'); +const assert = require('node:assert/strict'); +const fs = require('fs'); +const os = require('os'); +const path = require('path'); + +process.env.DATABRICKS_API_KEY = process.env.DATABRICKS_API_KEY || 'test-key'; +process.env.DATABRICKS_API_BASE = process.env.DATABRICKS_API_BASE || 'http://test.com'; +process.env.LOG_FILE_ENABLED = 'false'; + +// BudgetManager's constructor resolves its DB path from process.cwd()/data — +// run the whole test from a temp cwd so nothing touches the repo's live DB. +const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'lynkr-tpm-test-')); +const originalCwd = process.cwd(); +process.chdir(tmpDir); + +const { BudgetManager } = require('../src/budget'); + +let mgr; + +test.before(() => { + mgr = new BudgetManager({}); +}); + +test.after(() => { + process.chdir(originalCwd); + fs.rmSync(tmpDir, { recursive: true, force: true }); +}); + +test.beforeEach(() => { + delete process.env.LYNKR_TPM_LIMIT; + try { mgr.db.prepare('DELETE FROM token_rate').run(); } catch { /* fresh db */ } +}); + +test('TPM limiting is off unless LYNKR_TPM_LIMIT is set', () => { + const check = mgr.checkTokenRate('user-a', 1_000_000_000); + assert.equal(check.allowed, true); +}); + +test('a request whose estimate alone exceeds the limit is rejected', () => { + process.env.LYNKR_TPM_LIMIT = '10000'; + const check = mgr.checkTokenRate('user-a', 50_000); + assert.equal(check.allowed, false); + assert.equal(check.reason, 'token_rate_limit_minute'); + assert.equal(check.limit, 10000); + assert.ok(check.resetInMs > 0 && check.resetInMs <= 60_000); +}); + +test('recorded actual usage gates subsequent requests (true-up pattern)', () => { + process.env.LYNKR_TPM_LIMIT = '10000'; + assert.equal(mgr.checkTokenRate('user-b', 2000).allowed, true); + mgr.recordTokenUsage('user-b', 9000); + // Window now holds 9000 actual; a 2000-token estimate would breach 10000. + const check = mgr.checkTokenRate('user-b', 2000); + assert.equal(check.allowed, false); + assert.equal(check.current, 9000); + // A tiny request still fits. + assert.equal(mgr.checkTokenRate('user-b', 500).allowed, true); +}); + +test('windows are per-user — one user\'s consumption never gates another', () => { + process.env.LYNKR_TPM_LIMIT = '10000'; + mgr.recordTokenUsage('user-c', 9999); + assert.equal(mgr.checkTokenRate('user-c', 5000).allowed, false); + assert.equal(mgr.checkTokenRate('user-d', 5000).allowed, true); +}); + +test('the window resets after a minute', () => { + process.env.LYNKR_TPM_LIMIT = '10000'; + mgr.recordTokenUsage('user-e', 9999); + assert.equal(mgr.checkTokenRate('user-e', 5000).allowed, false); + // Age the window artificially. + mgr.db.prepare('UPDATE token_rate SET minute_window_start = ? WHERE user_id = ?') + .run(Date.now() - 61_000, 'user-e'); + assert.equal(mgr.checkTokenRate('user-e', 5000).allowed, true); +}); + +test('recordTokenUsage is a no-op when limiting is disabled', () => { + mgr.recordTokenUsage('user-f', 5000); + const row = mgr.db.prepare('SELECT * FROM token_rate WHERE user_id = ?').get('user-f'); + assert.equal(row, undefined); +});