CHANGELOG.md

Changelog

All notable changes to ThinkWatch are recorded here. Format follows Keep a Changelog; from 1.0.0 onwards versioning follows SemVer.

Until 1.0.0, any 0.y bump may include breaking changes — see the notes under each release. Operators upgrading inside the 0.x line should read the section for every intermediate version, not just the target.

Unreleased

[3.9.1] — 2026-10-23

A conversation whose history a provider encrypted, such as a Codex conversation over OpenAI's Responses API, now stays with that provider instead of failing on every turn once it moves, and a provider that cannot read such history says why. The thinkwatch-core crates move from v0.77.0 to v0.81.0.

No setting, schema or API route changes.

Fixed

• A conversation whose history a provider encrypted goes back to that provider. OpenAI's Responses API with store: false (Codex always sends it) hands the client encrypted state that comes back every turn: reasoning, remote compactions, Codex's messages between agents, encrypted tool output. Only the account that encrypted it can read it, so a conversation that moved to another provider — after five idle minutes, a tripped breaker or a failover — was refused on every turn from then on with invalid_encrypted_content. The gateway now remembers, for 30 days and whatever the affinity settings say, the provider that last answered each conversation (the authenticated user and the request's x-session-id, or else the session header the client sends by itself, such as Codex's session_id; over HTTP and over the WebSocket). The log row's session_id still comes from x-session-id only. A Responses request carrying encrypted content that cannot be left out goes only to that provider: no strategy pick, no failover, and its open breaker does not stop it. When that provider has no route for the model any more, the request is refused up front with 409, naming the provider and saying to start a new conversation. A request carrying only encrypted reasoning goes to that provider first and still fails over. Requests that name no conversation are routed as before. The request's log row records "pinned_by": "encrypted_history" when the provider was picked this way.
• A provider that cannot read a conversation's encrypted history says so. Its refusal, as a 4xx answer or as the error event of a stream that opened with 200, now reaches the caller and the log as SealedHistoryRefused: the provider cannot read this conversation's encrypted history because another provider encrypted it; route the conversation back to that provider or start a new conversation. The upstream's own words follow. It no longer counts against the route's breaker.

3.9.0 — 2026-10-20

The Add to ThinkWatch Lite link from a rotated key now carries the key's request limits. A database failure during authentication answers 503 or 500 with the usual JSON error instead of an empty 500, and password hashing no longer runs on the request workers. Monitoring that matches only on 500 sees a database outage during authentication as a 503 from this release on. The thinkwatch-core crates stay at v0.77.0.

No setting, schema or API route changes.

Added

• The rotated-key dialog's Add to ThinkWatch Lite link carries the key's request limits. The key's own enabled, unexpired AI-gateway rules that count requests over a window of whole minutes up to 60, or whole hours up to 24 (the console's 1 minute, 5 minutes, 1 hour, 5 hours and 24 hours among them), go into the link's limits parameter as Lite's rolling windows of the same length; of two rules on one window the smaller count goes, and at most 16 windows, those allowing the fewest requests per second. Token rules stay out, since they count weighted tokens and Lite counts plain ones; so do sliding 7-day rules (Lite's week is a calendar week), budgets, MCP rules and the owner's limits. A link with limits needs ThinkWatch Lite 2026.10.26 or later, which the dialog states under the button. Reading the rules needs rate_limits:read in a scope covering the key; without it, or when the rules cannot be read, the link is built without limits as before. A new key has no limits of its own yet, so its link carries none.

Fixed

• A database outage during authentication is a 503 with an error body, not an empty 500. When the session or API-key check could not reach Postgres (most often a connection-pool timeout), the console and the gateway answered 500 Internal Server Error with no body and, for the permission lookup, no log line. These lookups now answer like every handler does: 503 Service Unavailable with the usual JSON error when the database is unavailable, 500 with it for any other database error, and an error log naming the lookup that failed.
• Password hashing no longer holds up other requests. Logins, password changes and resets, new users, turning off two-factor authentication and first-run setup ran Argon2 (tens of milliseconds of CPU by design) on the server's async workers, so every other request scheduled on that worker waited for it. They now run it on the blocking thread pool, at most one run per CPU at a time as before, so a burst of logins holds no more memory than it did (19 MiB per run). The hashes and the constant-time login are unchanged.

3.8.3 — 2026-10-20

Tool-call inspection in streamed answers now decides only on arguments that have settled, and a hit that is recorded carries the whole call. The thinkwatch-core crates move from v0.75.0 to v0.77.0. The web image is built with Node.js 24.22.0 and pnpm 12.14.0.

No setting, schema or API route changes.

Changed

• The thinkwatch-core crates move from v0.75.0 to v0.77.0.
• Since a recorded hit now waits for the complete call, a streamed tool call that never completes (the upstream stream breaks off, or the client leaves, in the middle of it) leaves no tool-inspection row. Such a call was never whole on the client side either. Rules that cut still check the arguments while they arrive.
• The web image's build stage uses Node.js 24.22.0 and pnpm 12.14.0. The served image (nginx) is unchanged.
• The development compose file (deploy/docker-compose.dev.yml) runs Zitadel v4.19.5.

Fixed

• Tool-call inspection no longer judges arguments that are still arriving. A streamed tool call was checked against its arguments as far as they had come in. A host cut off in the middle of its name (http://127 before .0.0.1) looked external, and a pattern anchored at the end could match the half-received tail. A hit now counts only once it is settled; rules that cut still act as early as possible.
• A recorded hit carries the whole call. A hit that is only recorded, and every hit in observe mode, was recorded with the arguments as they stood at the first match. It is now recorded once the call is complete. A Chat Completions call with no finish_reason or [DONE] is checked when the stream ends.
• -T is an upload only in curl. The upload rule took any -T for curl -T, so docker compose exec -T next to a URL was recorded as a file upload.

3.8.2 — 2026-10-16

Tool-call inspection now checks every choice of an answer, not only the first, and redacted values are put back in every choice of a streamed answer. The thinkwatch-core crates move from v0.74.0 to v0.75.0, and rustls moves to 0.23.46 for four security advisories.

No setting, schema or API route changes.

Security

• Tool-call inspection checks every choice of an answer. An answer can carry several choices (Chat Completions choices, with n above 1) or candidates (Gemini candidates), and a streamed chunk several of them at once. Inspection looked only at the first element of each streamed Chat Completions chunk's choices, and only at the first Gemini candidate, streamed or whole. A tool call in any other choice or candidate reached the client uninspected: it was not recorded, and in enforce mode it was neither cut nor refused. Every choice and every candidate is now inspected. The arguments of a streamed Chat Completions call are put together both by the choice's index and by the choice's position in each chunk, the two ways clients assemble them, and both are checked. Whole Chat Completions answers already had every choice inspected. This holds whether the route forwards the answer in the caller's format or converts it.
• rustls 0.23.46, for RUSTSEC-2026-0346 to RUSTSEC-2026-0349.

Fixed

• Redacted values are put back in every choice of a streamed answer. With outbound redaction in enforce mode, a streamed Chat Completions answer with several choices, or a Gemini answer with several candidates, could reach the client with a placeholder still in it, or with the start of a placeholder from one choice in the text of another. Values were put back only in the first choice of each chunk, and the text held back while a placeholder might still be completing was kept for the answer as a whole rather than for each choice, so one choice finishing released what another was holding. Each choice is now restored on its own, and released when it finishes.

Changed

• thinkwatch-core crates (tw-bedrock, tw-breaker, tw-dialect, tw-guard) v0.74.0 → v0.75.0. Only tw-guard changes.

3.8.1 — 2026-10-13

Outbound redaction searches only the text written in a request, located by the structure of the request's format, and no longer changes ids or other values the protocol needs as they are. The thinkwatch-core crates move from v0.73.0 to v0.74.0.

Read before upgrading

• Only the written text of a request is searched. Fields outside it, such as metadata, user, tool parameter schemas, response format schemas and stop sequences, and blocks or fields the gateway does not recognise, are no longer searched: a value there goes upstream as written and is not reported. Hits inside ids and other protocol values are no longer reported either, so fewer redaction events (gateway.redaction_replaced, gateway.redaction_flagged) may be written than before.
• Stored bodies are masked as before. With audit.body_redact_pii on, stored request and response bodies are still searched over their whole text, fields outside the written text included, and so is the answer a log row stores under a content rule that checks answers.

Fixed

• Redaction no longer changes ids or protocol values. Before a request left, the built-in rules searched the whole body, ids included, and custom rules skipped only some identifiers, by key name. With outbound redaction On and the Chinese mainland mobile number rule (cn-mobile-phone, off by default) enabled, a Codex reasoning item id (rs_…) holding a run of 11 digits was taken for a mobile number and replaced with a placeholder. The upstream refused the request, and since Codex sends its reasoning items back on every turn, every later turn of that conversation was refused as well. Under Monitor the id was reported as a hit.

  The rules now search only the text written in the request: system prompts, the messages of every role (earlier answers and text documents included), reasoning text and summaries, tool-call arguments and inputs, tool results and tool descriptions, as well as Chat Completions' predicted output and the code a Gemini model executed and its output. Ids, types, roles, names, the model, signatures, encrypted content, media and settings are never searched. Each hop is searched in the format it is sent in: the caller's when the request is forwarded, the upstream's when it is converted.

Changed

• thinkwatch-core crates (tw-bedrock, tw-breaker, tw-dialect, tw-guard) v0.73.0 → v0.74.0.

3.8.0 — 2026-10-13

The security page is organised by what each guard protects against: Secrets & privacy (outbound redaction), Dangerous commands (tool-call inspection) and Content filter, after a Log of what they found. Each guard is On, Monitor or Off, and every rule has an action of its own. Content rules can now check the text of answers as well as requests, and can replace what they match with a text of their own. The thinkwatch-core crates move from v0.72.0 to v0.73.0.

Read before upgrading

• Policies saved before 3.8.0 keep their behaviour. Every new key is optional, and a policy without it behaves as before: redaction rules replace with placeholders, content rules check requests only and ignore case, and answers are not changed. The console leaves each new key out while it has its default, so a policy saved from the 3.8.0 console without using a new option stays readable by 3.7. Two redaction fixes apply to every policy (see Fixed).

• Upgrade every replica before using the new options, and do not roll back afterwards. 3.7 refuses a policy with keys it does not know. A 3.7 instance that reads a security.redact policy with actions or a custom rule's action, or a security.content policy with a custom rule's scope, case_sensitive or replacement, or the action replace, logs Unreadable guard policy and runs that guard on its factory policy: monitoring, built-in rules as shipped, no custom rules. During a rolling upgrade, change a rule's action, what it checks, its case or its replacement only once no 3.7 instance is left; to roll back after doing so, first remove those keys.

• New names in the console. Outbound redaction is now Secrets & privacy, and tool-call inspection is Dangerous commands; the content filter keeps its name. Every guard's modes are On, Monitor and Off; they were Off, Observe and a third named for the guard (Replace, Cut off, Enforce). On the role pages the permission groups pii_redactor and content_filter are shown as Secrets & privacy and as Content filter & dangerous commands. The settings keys (security.redact, security.inspect_tools, security.content), the mode values (enforce, observe, off), the permissions and the existing audit actions keep their names.

• More guard events reach the audit log, and with it the log forwarders. Hits found while a guard monitors were already audit events and still are. New in 3.8.0:

  • gateway.content_replaced, a new action, for text a content rule replaced.
  • Rules that check answers write content events too, with direction: response, part: response_text and the upstream in resource (provider:<name>); a replacement's event carries the rule's replacement. A request's events have direction: request.
  • A redaction rule whose action is to record writes gateway.redaction_flagged while the guard is On, where that action used to mean the guard was monitoring. Events are per rule, so one request can have both a gateway.redaction_replaced and a gateway.redaction_flagged event.
  • Content, redaction and tool-call events carry the guard's mode (observe for hits recorded while monitoring), and redaction events the rule's action (placeholder or record).
  • gateway_content_matched_total has a direction label.

  Check alerts and SIEM filters that key on these actions.

• Filtering answers changes what clients receive, within these limits. Under Monitor, or when every rule that checks answers only records, nothing is held back or changed. Under On, with a rule that deletes or replaces:

  • A streamed answer is never held back whole. Each text holds back only the last characters that could still grow into a match: one fewer than the longest keyword, or a regular expression's longest possible match, at most 256 characters, one more for a regular expression that looks at the next character (\b, $). Of a longer match the beginning may already have been sent. When a rule deletes (strip, or replace with an empty text), as many characters again are held back; deletions at the very start of a text do not add to that. Text held back is sent when the stream ends, breaks off or carries an error from the upstream.
  • Deleting is repeated until no match is left: text that comes together where something was deleted is checked again (sesecretcret becomes empty, not secret). A replacement is never checked again. A whole answer always comes out clean, and a streamed one comes out the same as the whole one wherever it is split, as long as a match is nested no more than one level deep. Of a match nested deeper the beginning may already have been sent; the rest of it is deleted, so the client never receives a whole match, except a regular expression that looks at the next character whose match lies entirely in text already sent. After 64 further passes, what is still changing is dropped from the answer.
  • Regular expressions anchored to the start or end of a text or a line (^, $, \A, \z) are checked once, on the model's own text, and rules that only record see the model's own text, not what deletions leave.
  • The frames that repeat the whole text (response.output_text.done, response.output_item.done, response.completed) say what the client received.
  • An answer that is not valid UTF-8 is read with replacement characters in place of the invalid bytes, and sent that way where the filter rewrites it; an element of a Gemini JSON-array stream that cannot be parsed is dropped.

• Logprobs are removed while an answer rule can change text. Under On, with a rule that checks answers and deletes or replaces, logprobs are removed from every streamed answer, whether or not the rule matched, and from a whole answer whose text changed, since they repeat the text token by token: Chat Completions' logprobs becomes null, the Responses API's logprobs an empty list, and Gemini's logprobsResult is left out.

• With a rule that checks answers, the log row stores the answer as the client received it. Its text has been through the content rules, and the caller's redacted values are masked again: by the redaction rules, and wherever a value the request's placeholders stand for appears as written, so a value its rule recognises only in context (a connection string's password) is masked even when the model repeats it alone. In a body that is valid JSON this masking changes only the contents of strings (also values written escaped inside them, up to six levels deep), never numbers, literals or structure, so the stored body stays valid JSON. Stored text that is not strictly valid JSON (an event stream, a plain-text error) is masked wherever a value appears, numbers included. Where values overlap, the longest one at each place is masked. Without such a rule the row stores the answer as before, with the placeholders the upstream returned. The response cache keeps the answer unfiltered, in placeholder form, either way, and the rules in force run again on every cache hit.

Added

• Answer text filtering. A custom content rule says what it checks: requests (the caller's messages and tool results), answers, or both (scope: [request], [response] or [request, response]; left out: requests). On an answer the rule reads the text the client is shown and the input of freeform tools such as Codex's apply_patch, not tool calls and not reasoning. It applies to whole answers, streamed answers on every API surface, answers served from the cache and the Responses WebSocket (each turn filtered as a stream of its own), in the client's format, after redacted values are restored, so a rule never cuts into a placeholder, and before tool calls are inspected. A rule that checks answers cannot refuse, and cannot match by code points; such a policy is refused when saved. See Read before upgrading for how streamed answers are held back and what the client and the log receive.
• Replace. A content rule can replace what it matches with a text of its own (action: replace, replacement, *** when left out, at most 200 characters), in requests and in answers. A replacement is not put back, and is not checked again. In a request, deleting and replacing are repeated until no match is left (a keyword split by hidden characters is caught once they are gone); a part still changing after 64 rounds is emptied.
• Case-sensitive content rules (case_sensitive: true), as redaction rules already were.
• Record-only redaction rules. A redaction rule's action is to replace with a placeholder or to record only: built-in rules through security.redact.actions ({"email": "record"}), custom rules through action. A recorded value goes upstream unchanged, and the hit is reported as gateway.redaction_flagged; as before, a custom rule's event carries only counts and a built-in rule's event a few masked values. With audit.body_redact_pii on, the value is still masked in stored bodies. Where a record-only match overlaps a match that is replaced, the value is replaced.
• Console. The security page has a Log tab (for holders of logs:read_all) listing the guards' audit events, filtered by guard, with the rule, where it matched (request, tool result, answer text, tool call), what was done and a link to the trace. Each guard tab shows its state with a dot (green On, amber Monitor, grey Off), one line on what the state does, and how many rules are on. Rules are grouped, with a checkbox per group that switches every rule shown in it in one save, and a search over names and patterns; each row has its switch and its action, chosen in place. While a guard monitors, each action is shown with "Monitoring". The content rule dialog sets what the rule checks, how it matches (with a Case-sensitive switch), its action and its replacement text; options that do not apply to answers are shown but unavailable, with the reason, and switch with a notice when the scope changes. Samples can be tried as a request or as an answer (POST /api/admin/security/content/test takes scope and replacement). The redaction rule dialog's Case-sensitive is a switch, and the dialog sets the rule's action.
• GET /api/audit/logs takes several actions in action, separated by commas; the first 32 are used.

Fixed

• Overlapping redaction matches no longer leave part of a value in clear. Where two rules matched overlapping text, only the first match was replaced, and the part of the other outside it went upstream as written. Overlapping matches now become one placeholder. Matches that do not count (on the gateway's own placeholders, in base64 payloads, a custom rule's match in a protocol identifier) are set aside before overlaps are resolved; before, a value that overlapped one of them was dropped with it and sent as written.
• A custom redaction rule written as a regular expression is matched within one JSON string at a time. A pattern such as \S+ could start on the request's structure, replace part of it and leave the request that went upstream invalid JSON, missing the value it was written for.

Changed

• thinkwatch-core crates (tw-bedrock, tw-breaker, tw-dialect, tw-guard) v0.72.0 → v0.73.0.

3.7.1 — 2026-10-12

Tool-call inspection no longer crashes on a URL written right after a non-ASCII character, which cut streamed answers off, and redacted values come back exactly as the caller sent them. The thinkwatch-core crates move from v0.71.0 to v0.72.0.

Fixed

• A tool call with a URL right after a non-ASCII character no longer breaks the answer. Tool-call inspection, on in observe mode by default, looks for URLs in the arguments of the tool calls an upstream returns. When a non-ASCII character stood directly before a URL, as in 文档见：https://…, finding the URL's scheme cut that character in two and the gateway panicked. A streamed answer stopped partway, with no error event and no end of stream, and the request was logged with status 499 and client_cancelled, as if the client had left; on the Responses WebSocket the whole connection closed. A whole answer was refused with a 500 whose body is Service panicked, and logged with status 499 as cancelled before the response. No tool-call event was written for that call. Rust's panic message (… is not a char boundary …, at tw-guard's tools/net.rs) went to standard error, and for a whole answer the gateway's log also has a Service panicked error. Affected since 3.0.0.

• Redacted values come back exactly as the caller sent them. With outbound redaction in enforce mode, a value is now remembered as it reads rather than as the request's JSON wrote it, and is escaped for each place it goes back into. A value holding characters that JSON writes with an escape, such as the line breaks of a private key, a quote or a backslash, came back wrong:

  • In a streamed answer, put back into text (the answer's own, or the input of a freeform tool such as Codex's apply_patch), a private key arrived with \n in place of each line break.
  • In a whole answer, including one served from the cache, such a value inside a tool call's arguments, which are JSON written into a JSON string, was escaped once instead of twice, so the arguments the client received did not parse, or parsed to a different value.

  A placeholder split across two pieces of a freeform tool's streamed input (response.custom_tool_call_input.delta) also reached the client as written; it is now put back.

Changed

• thinkwatch-core crates (tw-bedrock, tw-breaker, tw-dialect, tw-guard) v0.71.0 → v0.72.0. Only tw-guard changes.

3.7.0 — 2026-10-12

Outbound redaction keeps away from the structure of a request: it no longer looks inside field names, and custom rules leave alone the identifiers a protocol needs as they are, so a rule can no longer break the ids a client sends back with its history. A custom redaction rule now says whether it looks for its text as written or for a regular expression, and whether case matters. The thinkwatch-core crates move from v0.70.0 to v0.71.0.

Read before upgrading

• Custom redaction rules saved before 3.7.0 now match their text as written, ignoring case. A custom rule in security.redact takes two new optional keys, match (contains or regex) and case_sensitive. A rule without match is looked for as plain text, in which ., *, \d and the like have no special meaning; a rule without case_sensitive ignores case. Until now every custom redaction rule was a case-sensitive regular expression, and no rule saved by 3.0 to 3.6 carries either key, including the rules an earlier upgrade converted from the old security.pii_redactor_patterns setting. A rule written as a regular expression still loads and nothing reports it, but it stops matching: what it used to replace is sent upstream as it is and, with audit.body_redact_pii on, stored unmasked. Nothing is converted automatically. Once every replica runs 3.7.0, open each custom redaction rule on the console's security page, set Match by to Regex for one written as a regular expression, and tick Case-sensitive where case mattered. A policy written by a script or through PATCH /api/admin/settings needs "match": "regex" and "case_sensitive": true on each such rule. Settings still in the format from before 3.0.0, converted on the first start of 3.7.0, become case-sensitive regex rules and keep matching as before.
• Upgrade every replica before editing redaction rules, and do not roll back afterwards. 3.6 refuses a redaction policy with keys it does not know. A 3.6 instance that reads a policy in which any custom rule has match or case_sensitive logs Unreadable guard policy and runs redaction on the factory policy: observe mode, the built-in rules as shipped, no custom rules, so nothing is replaced on the wire. Any change to outbound redaction saved from the 3.7.0 console, even a mode switch, writes match on every custom rule. During a rolling upgrade, edit redaction rules only once no 3.6 instance is left; to roll back to 3.6 after editing them, first remove match and case_sensitive from every custom rule in security.redact.
• Redaction no longer searches field names, and custom rules skip protocol identifiers. No rule looks inside a JSON key any more. Custom rules also leave alone the values of id, call_id, item_id, tool_use_id, tool_call_id, toolUseId, previous_response_id, response_id, type, role, name, namespace, model, status, object and phase. Built-in rules still search those values, and every other value is searched by every rule as before, including fields such as user_id or aws_access_key_id inside tool arguments. A custom rule meant to catch a value under one of those keys, such as a tool name or a model name, no longer does. The same holds for masking stored bodies (audit.body_redact_pii).
• The private-key rule replaces less. The header of a key block and its end line are now looked for within the same text (see Fixed). A header in one message and an end line in another are no longer replaced, along with everything between them, as one key.

Added

• Match by and Case-sensitive for custom redaction rules. The custom rule dialog for outbound redaction offers Contains and Regex, as the content filter's does, and a Case-sensitive checkbox. Contains looks for the text exactly as written; leading and trailing spaces count. The rule list says how each rule matches, such as "Contains x, ignoring case" or "Regex x, case-sensitive". Trying a sample in the dialog uses the same settings: POST /api/admin/security/redact/test takes match and case_sensitive. Code points remain a content filter matcher and are refused for redaction.

Fixed

• Custom redaction rules no longer break the ids a client sends back. Rules ran over the whole request body, field names and identifiers included, so a custom rule could match inside an id: Codex sends ids such as fco_ followed by a hexadecimal UUID, and a rule whose text can be spelled in hexadecimal turned one into fco_…<<TW_SECRET_1>>…. The upstream rejected the request, and since the history is sent again on every turn, every later turn of that conversation as well. See Read before upgrading for what is no longer searched.
• Private keys stay within one text. The private-key rule looked for the end of a key block anywhere after its header. A header without an end in one message, such as a file read halfway or code that spells the marker, joined up with an end line in a later message, and everything in between, earlier turns included, was replaced by a single placeholder.

Changed

• thinkwatch-core crates (tw-bedrock, tw-breaker, tw-dialect, tw-guard) v0.70.0 → v0.71.0.
• The Licensor in LICENSE is now the project's owner, named in the file, instead of "ThinkWatch Contributors"; the Business Source License 1.1 terms are unchanged.
• A first pull request to this repository asks its author to sign the Contributor License Agreement (CLA.md) once, by a comment on the pull request; CONTRIBUTING.md explains how.

3.6.0 — 2026-10-12

A key shown once can go straight into ThinkWatch Lite: the console's key screens offer Add to ThinkWatch Lite, and the Configuration Guide has a tab for it. Every user, not only administrators, now sees the gateway address an administrator configured, and the model listings name only the models the calling key may use.

Read before upgrading

• Model listings follow the calling key. GET /v1/models and GET /v1beta/models listed every routed model whatever the key; they now list only the models a request with that key is let through for (see Fixed). A client that lists models with a narrowed key sees fewer of them, and a key whose allow-list and owner's roles have nothing in common lists none. No setting, schema, Helm value or Redis key changes.
• general.public_host must be a host name or an IP address. Saving it used to accept anything without / or ://, such as gateway.example.com:8443, and the Configuration Guide then built a broken URL from it. Such a value is now refused when saved, and one already stored counts as not set: the guide uses the host the console is open at instead. If yours carries a port, move the port to general.public_port.
• The console's Content-Security-Policy allows frame-src thinkwatch:. The web image's nginx now sends frame-src 'self' thinkwatch:, which Add to ThinkWatch Lite needs to hand its link to the app (see Added). A policy of your own in front of the console needs the same directive, or the button does nothing.

Added

• "Add to ThinkWatch Lite" for a key shown once. The new-key and rotated-key dialogs have an Add to ThinkWatch Lite button next to Copy; at the end of first-run setup it is on its own line under the key. It opens a thinkwatch://import link with this gateway and the key, and ThinkWatch Lite (2026.10.16 or later) opens its New upstream dialog with them filled in; the upstream is created when the user confirms it there. The link is built in the browser from the key on screen and the gateway's public address, and is sent nowhere. It is opened in a hidden frame, so a browser without Lite installed keeps the page, and the key on it, instead of showing an error page in its place. The button is unavailable, with the reason beside it, for a key without the ai_gateway surface, for a user whose roles do not grant ai_gateway:use (a key is always its creator's, and the gateway would refuse its every request), and for a gateway whose public address Lite does not accept. Lite takes https://, and plain http:// only on this machine or a private network: private IPv4 ranges (10/8, 172.16/12, 192.168/16, 100.64/10, 169.254/16), private IPv6 ranges (fc00::/7, fe80::/10), and names with no dot or ending in .local, .lan, .internal, .intranet, .corp, .home.arpa or .localhost. Elsewhere, set general.public_protocol to https where the gateway is served over TLS. Once clicked, a line under it links to the download page. Phones get Copy only.
• The Configuration Guide has a ThinkWatch Lite tab, first among the AI gateway clients: what Lite does, and a button that opens the new-key dialog filled in for it (named "ThinkWatch Lite", AI gateway only). Users without api_keys:create are told how to get a key and add it by hand.
• GET /api/gateway/info on the console port gives any signed-in user the configured parts of the gateway's public address: {"public_protocol": "https", "public_host": "gateway.example.com", "public_port": 443, "gateway_port": 3000}. public_protocol, public_host and public_port are general.public_protocol, general.public_host and general.public_port, each null when not set (or when the stored value is not one the setting accepts); gateway_port is the port the gateway listens on. The console fills in what is not set from the address it is open at, the browser's scheme and host, and uses gateway_port when no public port is set. The server reads no request headers for it.

Fixed

• The Configuration Guide shows the configured gateway address to every user. It read the address from GET /api/admin/settings/system, which needs settings:read: everyone else got 403 and an address built from the browser's host and port 3000, whatever general.public_* said. It now reads GET /api/gateway/info. Until that has loaded, and if it cannot be read, the examples use the browser's scheme and host with port 3000, as before.
• GET /v1/models and GET /v1beta/models list only the models the calling key may use: the key's allowed_models intersected with what its owner's roles grant, matched exactly or by prefix, the check a request for the model meets. They listed every routed model, and a request for one outside that scope was then refused. A model named in that allow-list is listed when a route serves it, so a key narrowed to gpt-4o lists gpt-4o when the route is a prefix such as the gpt- route of a provider with no routes of its own.
• Saving general.public_host checks it (see Read before upgrading): PATCH /api/admin/settings answers 400, naming what the setting takes, for a value with a scheme, a port, a path, a user or anything else beyond a host name or an IP address. In Settings, the field's save marker turns to an error that gives this message.

Changed

• The Public Gateway URL hint in Settings says where an empty protocol or host comes from (the address the console is open at) and that port 0 is the gateway's listening port, and names Add to ThinkWatch Lite among what uses the address.

3.5.0 — 2026-10-11

Limits hold where they used to leak: route caps are enforced, a limits store that cannot be read follows security.rate_limit_fail_closed, a refused model or tool no longer spends a request, a team-scoped role no longer limits the gateway it does not open, and cached answers count their tokens. A client can ask the gateway what room its API key has left (GET /v1/usage), and an API key's own limits are edited in the console. The legacy monthly token quota is gone, and the thinkwatch-core crates move from v0.69.0 to v0.70.0.

Read before upgrading

• Route caps are enforced. A route's RPM cap and TPM cap (model_routes.rpm_cap / tpm_cap, set in the route editor) were saved but never applied. They now are, per route over a sliding minute, the TPM cap in weighted tokens like every other token limit (counted after the response). A route at either cap is skipped and the model's next route serves the request; when every route left is at its cap, the request gets 429 with Retry-After, labelled route:requests/1m or route:tokens/1m. Such a request has passed the caller's own limits and counts on them, as a request every upstream fails does. Review the caps already set on your routes: a cap that did nothing until now takes effect on upgrade. Routes with a cap write new Redis keys, ratelimit:{route:<route_id>}:ai_gateway:route:….
• A limits store that cannot be read follows security.rate_limit_fail_closed. When the database failed while a request's rate limits and budgets were being loaded, the request ran with no limits at all; when Redis failed while a budget was being read, the budget read as unspent. Both now follow the setting, as a Redis failure in the rate limiter always has, and so do the route caps: off (the default), the request goes on and a warning with the error is logged (gateway_limits_load_fail_open_total, lifecycle_budget_fail_open_total); on, the request is refused, labelled limits_unavailable, budget_unavailable or rate_limiter_unavailable (on the AI gateway, 429 with Retry-After: 30). With the setting off, a Redis error used to fail every AI-gateway request anyway, with 502 Quota exceeded, at the legacy quota check (see Changed); such a request now goes on, as the setting says. Turn the setting on to keep refusing requests while Redis is down.
• A role granted at team scope no longer limits gateway requests. Such a grant administers that team and has never opened a model or an MCP tool at the gateways, but its rate limits and budgets were applied to every gateway request of the user who held it. A role's limits now apply exactly where its gateway grant does: roles assigned globally, and roles attached to a team the user is a member of. A user who held a limiting role only at team scope is no longer held to its limits; assign the role globally, attach it to the team, or set a user limit if they were meant.
• Answers from the response cache count their tokens. A cache hit counted toward request limits but added no tokens to token limits and budgets, so repeating a cached request got round them. It now adds the tokens the cached answer records, weighted like any answer (all of its input as plain input). The cost a cache hit reports stays 0. Callers that lean on the cache reach their token limits and budgets sooner.
• Each API key and each user get day and month usage counters in Redis. An AI-gateway request made with a key now also counts on the key's usage:{user:<owner id>}:api_key_lineage:<lineage id>:daily:<date> and …:monthly:<month> and on its owner's usage:{user:<owner id>}:user:<owner id>:daily:<date> and …:monthly:<month>, hashes that expire two periods after their last write. That is one more Redis script per request once it passes the rate limits, and one when its tokens are counted, after the call or when it is answered from the response cache. The counters start empty, so the usage that GET /v1/usage reports counts from the upgrade on. No setting, database schema or Helm value changes.

Added

• GET /v1/usage tells a client what room its API key has left. Called on the gateway port with a gateway API key, as a model request is, it answers about one of two subjects, named in scope:

  • "key" when the key has limits of its own on the AI gateway: limits lists only the key's limits, each with used counted for the key, and usage is the key's own (across its rotations).
  • "user" when it has none: limits lists its owner's effective limits (roles, including those a team grants, merged most-restrictive, then the user's overrides), each with used counted for everything the owner does, and usage is the owner's total over all of their keys. With no limits on the owner either, limits is empty.

  {
    "scope": "key",
    "usage": {"requests_today": 12, "tokens_today": 48210,
              "requests_month": 340, "tokens_month": 1290455,
              "cost_usd_month": 3.82},
    "limits": [
      {"scope": "key", "kind": "tokens", "window": "daily", "window_secs": null,
       "limit": 100000, "used": 48210, "resets_at": "2026-10-12T00:00:00Z"},
      {"scope": "key", "kind": "requests", "window": "1m", "window_secs": 60,
       "limit": 60, "used": 4, "resets_at": null}
    ],
    "expires_at": "2026-12-31T00:00:00Z"
  }

  usage holds the requests the rate limits let through and the weighted tokens limits count, answers from the response cache included, for the UTC day and month, and the cost this month from the request log (null without ClickHouse). Every limit's used is read from the counter that refuses requests, and the one with the least left comes first. window is a rate limit's sliding window (1m, 5m, 1h, 5h, 1d, 1w, with its length in window_secs) or a budget's calendar period (daily, weekly, monthly, with its end in resets_at, UTC). Limits on the MCP gateway are not listed. expires_at is the key's expiry or the end of its rotation grace period, whichever comes first. A key a model request would refuse gets the same 401 or 403, and 503 means the key's limits or their counters could not be read, whichever way security.rate_limit_fail_closed is set. Calling it charges no limit, writes no request log row and is not a use of the key: last_used_at stays as it was, so polling does not keep an idle key from its inactivity timeout. Whoever holds a key without limits of its own sees its owner's totals and limits; a key handed to someone else should carry limits of its own. The console's Configuration Guide lists the endpoint with the other gateway endpoints.

• An API key's own limits are edited in the console. The key's edit dialog has a Limits tab: rate limits (requests or weighted tokens over 1m, 5m, 1h, 5h, 1d or 1w) and budgets (daily, weekly or monthly weighted tokens), each with what it has used, added, changed and removed through the existing limits endpoints. They follow the key across rotations. Reading needs rate_limits:read and changing rate_limits:write, in a scope that covers the key, as before; without write access the tab is read-only, and without read access the dialog is unchanged.

Fixed

• A request the key may not make no longer spends a request limit. The rate-limit check, which counts the request, ran before the model check on the AI gateway and before the tool check on the MCP gateway, so a refused model or tool used up the caller's requests limits. Access is now checked first (after the budget check, which counts nothing).
• Route caps are applied, limits and budgets that cannot be read follow the fail-closed setting, team-scoped roles no longer limit and cached answers count their tokens: see Read before upgrading.
• Comments that described a limits cache, a limits:changed subscriber and team-level budget caps, none of which exist, now say what happens: limits are read from the database on every request, nothing subscribes to limits:changed, and teams carry no limits of their own.

Changed

• The legacy monthly token quota is removed. Every request counted its tokens into quota:{<user or key>:<model>}:used:<YYYY-MM> in Redis against a limit nothing could set, so it never refused anything. The gateway no longer reads or writes these keys; existing quota:* keys can be deleted. Monthly limits are budgets. Its metric, gateway_quota_overflow_total, is gone too: the Grafana overview panel that plotted it shows routes at their cap (gateway_route_capped_total) instead.
• The security.rate_limit_fail_closed hint in Settings names everything the setting now covers.
• thinkwatch-core crates (tw-bedrock, tw-breaker, tw-dialect, tw-guard) v0.69.0 → v0.70.0. The four crates do not change beyond their version.

3.4.0 — 2026-10-11

Conversations with reasoning models behind Chat-format upstreams keep their reasoning from turn to turn, Gemini's tool loops keep their thought signatures through a conversion, and a tool call that a Chat-format model writes into its answer text reaches the caller as a tool call. The thinkwatch-core crates move from v0.68.0 to v0.69.0.

Read before upgrading

• Converted answers from Chat-format upstreams can carry tool calls that arrived as text before. Self-hosted and relayed Chat-format models (DeepSeek, GLM, Qwen, Llama and others) sometimes write a tool call into content in the template they were trained on instead of answering with tool_calls. On a route that converts the request to the Chat format, such a call to a tool the request defined now reaches the caller as a structured tool call, which the caller executes, where it used to be a tagged piece of prose the caller showed and did nothing with. Tool-call inspection sees these calls like any other and refuses or cuts them under the same rules; the request log's response body records them as tool calls, and a recovered call ends the answer with the tool-use stop reason instead of stop. A request that defined no tools, a name the request did not define, and a bare JSON object in prose stay text. A Chat caller on a Chat-format route is forwarded as sent and unchanged. No setting, schema, API route, Helm value or Redis key changes.

Fixed

• Gemini thought signatures are not counted as answer text. When a request's usage has to be estimated (usage_estimated), the thoughtSignature on a Gemini thought or function call — an opaque blob of up to several kilobytes — was counted as output text; it is left out now, as the other formats' reasoning signatures were.

Changed

• thinkwatch-core crates (tw-bedrock, tw-breaker, tw-dialect, tw-guard) v0.68.0 → v0.69.0. Only tw-dialect changes:
  • Reasoning from a Chat-format upstream round-trips (tw-dialect). Reasoning a Chat-format upstream returned (reasoning_content, or reasoning as OpenRouter and vLLM call it) was dropped from the next turn's request on every route, so a Claude Code or Codex conversation with tools against DeepSeek, which requires the reasoning of a tool-calling turn on the following requests of that turn, lost it every turn. Such reasoning is now carried in the caller's own format with a tw1.ch. signature, read back on the next turn, and written as reasoning_content only to a Chat-format upstream; to an upstream of another format it is left out as before, and reasoning another vendor signed is still not sent to a Chat-format upstream. Consecutive assistant messages are merged before a request is written in the Chat format, so a Responses caller's reasoning, text and tool call of one turn arrive as one message. The reasoning history a request carries back counts as its input, so the input tokens of those turns grow by it, as the upstream charges them.
  • Gemini thought signatures on function calls are kept (tw-dialect). Gemini with thinking puts a turn's thought signature on the first functionCall part, and the conversion only read signatures from thought parts, so a converted Gemini tool loop lost the signature and with it the model's reasoning continuity. The signature on a function call is now carried through a conversion and written back on the functionCall part; a call with no real signature gets the skip_thought_signature_validator placeholder as before. A Gemini request forwarded as sent that carries a signature from another route on a functionCall part keeps the part with the placeholder, where the part was removed and the tool loop broke.
  • Tool calls written into a Chat-format upstream's answer text are recognised (tw-dialect): see Read before upgrading. In this edition the answer assembled for the request log and the cache of a Chat caller's stream forwarded as sent to a Chat-format upstream keeps such a call as the text the caller received, rather than the structured call the conversion would have made of it.

3.3.1 — 2026-10-10

Streams are logged as they ended. A caller that hangs up once it has read the whole answer, as Codex does, finished the request; an error the upstream or Bedrock reports partway through a stream is logged with the status the same error has as an answer, and counts against the route only when that answer would; a Responses WebSocket turn ends at its last event. The thinkwatch-core crates move from v0.67.1 to v0.68.0.

Read before upgrading

• Request logs and error rates change for streams. Codex streams that were logged as cancelled (499) are logged as finished (200) and billed on the reported usage. Streams the upstream failed partway are logged with a 4xx or 5xx status_code instead of 200, so error rates that count status_code ≥ 400 include them; client_status keeps the 200 the caller received. No configuration, database schema, API or Helm value changes.

Fixed

• A caller that hangs up on the answer's last event finished the request. Codex closes the connection as soon as it has read response.completed, without waiting for the stream to end, and an upstream can end its stream a while after that event: the ChatGPT Codex backend often does, and so can a relay in front of it. A stream forwarded in its own format whose caller left in that window was logged as cancelled (499, stream_outcome: client_cancelled), though the caller had read the whole answer. Once the answer's last event has gone out — Responses response.completed, response.incomplete or response.failed, Anthropic message_stop, Chat [DONE] — a caller that leaves is logged as if the stream had run to its end: 200, billed on the usage the upstream reported, or, after response.failed, the upstream's failure (see the next entry). Converted streams were not affected: their last event is written once the upstream's stream has ended.
• An error the upstream reports partway through a stream fails the request. An upstream that has answered 200 can still report an error in the stream: an Anthropic error event (an overloaded_error arrives this way), Responses response.failed, an error in a Chat or Gemini chunk. When the stream then ended normally, the request was logged as a success (200) and counted as one by the route's circuit breaker. It is now logged as the error it would have been as an answer, with what the upstream said: stream_outcome upstream_error, client_status 200 (the status the caller's response went out with), and the status_code and error_type that answer would have had, read off the error's own type or code. A rate limit or a spent quota is 429 (UpstreamRateLimited); a context too long or another invalid request 400, an unknown model 404, an overload 529 or 503 (ProviderHttpError); an error that names no status 502 (ProviderError). The route's health and circuit breaker treat it as they treat that answer: a 5xx, a 429 or a refused credential counts against the route, so a route whose streams keep failing this way opens its breaker, while a request the upstream refuses, such as one past the model's context window, does not. This holds for streams forwarded as sent and converted ones, and for a caller that leaves after the error. Billing is unchanged: such a request is billed on the usage the upstream reported.
  • Error rates that count status_code ≥ 400 now include these requests.
  • The rows of other streams that failed after the response went out as 200 — broken off, cut by tool-call inspection, refused by the upstream before its first byte — carry client_status too.
• An exception Bedrock reports partway through a stream is logged as the same exception as an answer. Bedrock reports a failure in a stream it has started as an exception event: throttlingException when it throttles partway through, validationException, internalServerException, serviceUnavailableException and others. Each was logged as the stream breaking off in transit (502, error_type transport) and counted against the route's circuit breaker. Each is now logged with the status the desktop gateway gives that exception and with what Bedrock said: throttling 429 (UpstreamRateLimited), an invalid request 400, a model timeout 504, the service unavailable 503, any other exception 500 (ProviderHttpError). Access denied is logged as a refused credential (UpstreamAuthError), without Bedrock's words, which name the IAM principal, as for a 403 answer. The route's health and circuit breaker treat it as they treat that answer (see the previous entry), so a request Bedrock refuses as invalid no longer counts against the route. What the caller receives is unchanged, and a damaged event-stream frame is still logged as transport 502.
• A Responses WebSocket turn ends at its last event. On GET /v1/responses over a WebSocket, a turn lasted until the upstream's stream ended, and the connection took the next turn only then, so an upstream that keeps its stream open after response.completed held up every following turn by that long. The next turn now starts at once.

Changed

• thinkwatch-core crates (tw-bedrock, tw-breaker, tw-dialect, tw-guard) v0.67.1 → v0.68.0. tw-dialect changes what the gateway does, tw-guard only adds to its API, and tw-bedrock and tw-breaker do not change:
  • Long usage objects are read (tw-dialect). The usage reader skipped a usage object longer than 8 KB. The ChatGPT Codex backend's usage carries an attribution breakdown that grows with the conversation, so behind an upstream that passes it on, a longer Codex session's requests were billed on an estimate (usage_estimated) instead of the reported counts. Usage objects up to 1 MiB are now read. One cut across chunks is read on from where it stopped, nothing nested inside a usage object is counted as a second usage, and a stream that arrives in chunks shorter than the "usage" key no longer loses it.
  • A null error in a stream chunk is not an error (tw-dialect). Some OpenAI-compatible relays and Gemini-format upstreams write "error": null in every streamed chunk, and the Chat and Gemini stream readers took each such chunk for an error the upstream reported. A stream converted to another format ended in an error and lost its text, and for such a stream forwarded as sent, no assembled answer was cached or captured as the response body. A Gemini caller's tool result carrying "error": null and no output went to an upstream of another format as a failed one. A null error is now ignored, and a Chat chunk's error without a message reaches the caller of a converted stream as a fixed sentence instead of the whole chunk, which could hold answer text. The gateway's own check for an error partway through a stream (see Fixed) ignores a null error as well.
  • tw_dialect::convert::ends_answer recognises an answer's last event. The gateway uses it to tell a caller that left after the whole answer from one that left partway, and to end a WebSocket turn (see Fixed).
  • tw-guard adds, for the desktop gateway's security log, the locate module, content::places, content::places_text and RulePlaces, content::Rule::draw, Ledger::placeholder_of, RuleSet::custom_pattern, tools::Rule::find_all, and the fields pattern, arguments, capped, places and path on tools::wall::Verdict. Secret redaction, the content filter and tool-call inspection find, replace, refuse and cut exactly what they did before. Verdict::arguments is a flagged tool call's arguments with redacted values restored; the gateway does not log it, and its tool-call audit entries keep the fields they had.

3.3.0 — 2026-10-09

Requests converted for Claude now use prompt caching, and Codex works through routes that convert its requests to another format: its tools, its history and its compactions arrive. A Responses request that only OpenAI can read — one that continues a conversation kept on OpenAI's servers, or carries a compaction OpenAI wrote — now goes to a route that speaks Responses instead of being refused. The thinkwatch-core crates move from v0.65.0 to v0.67.1.

Read before upgrading

• Converted requests to Claude are cached, and billed as cache traffic. A request converted for an Anthropic-format upstream, or for a Claude model on Bedrock that AWS lists for prompt caching (Claude 3.5 Sonnet v2, Claude 3.7 Sonnet, and Claude 4.5 and later), from a client that marks no cache breakpoints — Codex, Chat and Gemini clients, and an Anthropic client whose request goes to Bedrock without cache_control — now gets them at the end of the tools, the system prompt and the last two user turns, with the 5-minute lifetime.
  • The upstream bills the prefix a request writes at its cache-write rate (1.25× input at Anthropic) and what the next turns read back at its cache-read rate (0.1×). ThinkWatch prices and counts them the same way: cost, budgets and tokens rate limits weigh a cache write by the model's cache_write_weight and a read by its cache_read_weight, 1.25× and 0.1× its input weight when unset. Set them on the model to match what your upstream charges.
  • A conversation of several turns costs less. A one-off request with a long prompt costs up to a quarter more on its input.
  • Prompt token counts (input_tokens on the log rows, quotas) include cached input in full, as before.
  • Requests that mark their own breakpoints, such as Claude Code's, and requests forwarded in their own format are unchanged.
  • An upstream that refuses the added breakpoints — a 400 that names cache_control, cachePoint or prompt caching, as an Anthropic-compatible endpoint that does not know them answers — gets the request once more without them, and later requests to it for that model leave them out from the start. Each instance remembers this until a provider or model is changed or it restarts. Breakpoints a client marked itself are never taken out, and the probes that check a route's API format go without breakpoints.
• Rolling back with Codex sessions. A Codex session that compacted through a converted route on 3.3.0 carries a compaction that 3.2.1 refuses, so after a rollback, or on a 3.2.1 instance during the rollout, it cannot continue and needs a new session.

Nothing else needs action: no setting, schema, API route or Redis key changes.

Fixed

• Responses requests that only OpenAI can read. A Responses request that continues a conversation kept on OpenAI's servers (previous_response_id, conversation, prompt, background), points at a stored item (item_reference) or carries a compaction OpenAI wrote was refused with 400, even on a route that speaks Responses and would have forwarded it as sent: every request was decoded for its usage estimate, and one that could not be decoded was refused. Codex sends such a compaction in every request after compacting a session through an OpenAI upstream, so the session could not go on; and a WebSocket turn naming a response other than the connection's last one, meant to go upstream as sent, was refused too. These requests now go to the model's routes that speak Responses, and only a model with none refuses them, with the reason. Their input estimate counts the request's text.

Changed

• thinkwatch-core crates (tw-bedrock, tw-breaker, tw-dialect, tw-guard) v0.65.0 → v0.67.1. Only tw-dialect, the format conversion, changes:
  • Prompt caching for Claude on converted requests (see Read before upgrading).
  • Codex tools. For some models Codex declares every tool in an additional_tools input item and sends no top-level tools. Those items were dropped, so a route converting to Anthropic, Chat, Gemini or Bedrock sent no tools and Codex could not read files or run commands. Every tool now arrives, and tool calls come back under the names and kinds Codex declared. Local shell calls, tool search, reasoning-effort changes and agent messages in the history are converted instead of dropped.
  • Codex compaction on converted routes. Codex compacts a long session through a provider named OpenAI by asking for exactly one compaction item, which a converted route could not give, so compacting failed. The upstream now writes a handoff summary, Codex receives it as its compaction, and later requests carry it back. The summary request is billed like any other. The summary travels in the item's encrypted_content (tw1.c.…) base64-encoded, not encrypted; with audit.body_redact_pii on, captured bodies have it redacted like the rest of the body. OpenAI's own compactions still cannot be read by an upstream of another format, and are refused saying so.
  • System messages in mid-conversation stay in place. A system or developer message after the conversation has started was added to the system prompt on a converted route. It now stays where it was given: a developer message for a Responses upstream, and a user turn wrapped in <system-reminder> for Anthropic, Chat, Gemini and Bedrock upstreams. The system prompt stays the same from one turn to the next, which is what keeps the prompt cache working.
  • verbosity (Chat) and text.verbosity (Responses) carry across the conversion to GPT-5 and later models. Other models still leave it out.

3.2.1 — 2026-10-09

The Helm chart and the Compose file now pull the published images: they named ghcr.io/thinkwatch/…, which this project does not publish to, so a default install could not pull. A TOTP code now logs in once, and log rows are no longer lost while ClickHouse is unreachable. Five reads from ClickHouse returned wrong or missing data and now return what is stored: the SLO latency percentiles, a route's latency history, the times on traces and log bodies, a trace's app events, and the log lists for a range written with a T; and a failed read is now an error or a logged warning instead of an empty answer. The server also reads from ClickHouse 26.9 and later, the console shows its own version again, and the dependencies move to current releases, several of them for security advisories. The thinkwatch-core crates move from v0.62.0 to v0.65.0, which makes outbound redaction and the content filter faster without changing what they find.

No setting, schema or API route changes, and instances of 3.2.0 and 3.2.1 can run side by side during a rollout.

Read before upgrading

• Image names in your own files. A values file or --set flag of your own that names ghcr.io/thinkwatch/think-watch-server or …-web keeps overriding the chart's new default, and a copy of deploy/docker-compose.yml keeps the old name: change them to ghcr.io/thinkwatchproject/…. A pinned image tag has no v: 3.2.1, not v3.2.1.

Fixed

• Image names. The Helm chart (image.server.repository, image.web.repository) and deploy/docker-compose.yml named ghcr.io/thinkwatch/think-watch-server and …-web, which this project does not publish to and which refuses anonymous pulls, so helm install with the default values and docker compose pull failed. Releases are published to ghcr.io/thinkwatchproject/think-watch-server and …-web, and both files now name them. The tag examples in the chart's README and values-production.yaml.example wrote v0.2.0; published tags have no v, and the examples now say 3.2.1.
• SLO latency percentiles. GET /api/admin/slo reported p50, p95 and p99 latencies next to zero: the percentiles kept the column's integer type and were read as floating-point numbers. They now give the latencies in milliseconds.
• Route latency history. The latency sparkline on the routing page (GET /api/admin/models/{model_id}/route-history) was always empty: the query's column types differed from the ones the server read, and the failure fell back to no data. It now shows each minute's p50 and p95, and a gap for a minute whose requests all failed before a latency was recorded.
• Times on traces and log bodies. The trace view and the gateway and MCP log body views put the month name where the minutes belong (2026-10-06T04:October:52), because %M in ClickHouse's formatDateTime is the month name. They now read 04:31:52.
• App events on traces. A trace never listed app log events: the query that finds them compared a text column with a time, ClickHouse refused it, and the refusal showed as no events.
• ClickHouse 26.9 and later. ClickHouse 26.9 compresses query results with ZSTD unless the client names a method, and the server's client reads only LZ4, so every read failed with incorrect magic number. The server now names LZ4 on every query, which every ClickHouse version supports. Query results are also checked against the types the server reads them into, so a query whose types drift fails instead of returning garbage; every query was checked against ClickHouse 26.8 and 26.9.
• Log rows kept through a ClickHouse outage. The log writer is meant to keep a batch that ClickHouse did not take and retry it on the next tick, up to 1000 entries per table. A batch that failed while being sent — ClickHouse restarting or unreachable, the usual case — had already been emptied, so its gateway, MCP, audit, access or app log rows were lost while the server logged retaining N entries for retry. They are now kept and retried. A batch that ClickHouse stored but whose answer never arrived is written a second time.
• Time ranges on the log lists. On the gateway, MCP, audit, app and access log lists, a from written YYYY-MM-DDTHH:MM:SS, the form the API's error message suggests, dropped every row of that day from items while total still counted them: the list compared the times as text. The console sends YYYY-MM-DD HH:MM:SS and was not affected.
• ClickHouse failures no longer read as no data. The trace view, the usage-license meter (/api/admin/usage-license) and the per-period series and previous-period totals of the usage and cost statistics turned a failed ClickHouse query into an empty or zero answer, with nothing in the log. They now answer 500 and log the error. The dashboard's request and active-key tiles and the call counts on the MCP server list still answer without the part that failed, so the page loads, and now log a warning naming the query.
• The console's version. The sidebar of the web console showed v0.0.0 in every released web image, because the image build could not see the version file it reads. It now shows the release.

Security

• A TOTP code logs in once. A code accepted at login could log in again, with the password, for as long as it stayed valid: up to 90 seconds with the step of clock drift allowed either side, so a code read over a shoulder or a shared screen was enough. The last time step accepted for each user is now kept in Redis (totp_last_step:<user id>, for three minutes), and a code whose step is not later is refused, the code that turned TOTP on included. Logging in twice within one 30-second step needs the next code; one step of clock drift is still accepted. A refused replay is audited as auth.totp_failed with "replayed": true. Instances of 3.2.0 do not check, so this holds once the rollout is done.
• rand 0.10.3 (RUSTSEC-2026-0097) and xxhash-rust 0.8.19 (GHSA-6g2r-675j-hx59). Neither affects how ThinkWatch uses the crate: it installs no log logger that draws random numbers, and it hashes with xxh3's default secret only.
• The web console's build tools: brace-expansion 2.1.7 and 5.0.12, and ip-address 10.7.3 (Dependabot alerts on dependencies of eslint and the shadcn CLI). None of them is part of the console that is served.

Changed

• thinkwatch-core crates (tw-bedrock, tw-breaker, tw-dialect, tw-guard) v0.62.0 → v0.65.0. Secret scanning and the content filter do much less work per request: they search bytes directly, skip plain ASCII where no rule can match it, and lowercase text in runs. Core tests each fast path against the code it replaced on randomly generated text, so they find exactly what they found before.
• The bundled ClickHouse (Helm clickhouse.image, deploy/docker-compose.yml) moves from 26.3 to 26.8, both long-term-support releases. An upgrade restarts it on 26.8 with its data. ClickHouse does not promise that an older version reads what a newer one wrote, so to roll back to 3.2.0 later, keep clickhouse.image at 26.8.
• Dependencies move to current releases, among them sqlx 0.9, clickhouse 0.15, tower-http 0.7, utoipa 6, jsonwebtoken 11, argon2 0.6, aes-gcm 0.11, p256 0.14 and totp-rs 6. Password hashes, encrypted secrets, issued tokens, request signatures and TOTP codes from 3.2.0 keep verifying, and what 3.2.1 writes verifies on 3.2.0; known-answer tests cover each.
• Tests now hold two things that could otherwise break unnoticed: a login for an address with no account still spends the time of a password check (so its timing does not tell which addresses have accounts), and a column added to a ClickHouse log table comes with a default (without one, the previous release's inserts into that table fail during a rolling upgrade or after a rollback).
• The server image is built with Rust 1.99, and the web image serves from nginx 1.31 (was 1.29), built with Node 24.21 and pnpm 12. Building from source needs Rust 1.94.1 or newer (was 1.85).

3.2.0 — 2026-10-05

Redis Cluster and Redis over TLS now work, so managed Redis services can be used. Usage limits work as documented: token limits refuse, an API key's limits count on that key, and a refused request counts against nothing. Server instances can start together against one database, a restart no longer clears captured bodies older than 30 days, and the Helm network policy allows the ports the database URLs name. The thinkwatch-core crates stay at v0.62.0.

Read before upgrading

• Rate-limit windows start empty. Rate-limit counters move to new Redis keys (one hash per counter, tagged so that Redis Cluster can run them), and the counts from before the upgrade are not carried over: every window starts empty and fills from the first request after the upgrade. The old keys expire by themselves within two window lengths. Users' budget counters are kept; API keys' are not (see below).
• Route health starts fresh. A route's samples, circuit breaker and lifetime request count move to new keys for the same reason, so every route starts closed with nothing counted. The old lifetime counters never expire; redis-cli --scan --pattern 'route_health:[0-9a-f]*' | xargs redis-cli del removes them (the new keys start route_health:{).
• An API key's own budgets start again. 3.1.0 counted a key's budget on its owner's counter (budget:user:<user id>:…); 3.2.0 counts it on the key's own (budget:api_key_lineage:<lineage id>:…), which starts at zero. A key with a monthly budget of its own can spend all of it again in the rest of the month.
• Limits are looser while the rollout runs. Pods of 3.1.0 and 3.2.0 count rate limits and route health on different keys, so each sees only its own version's requests: limits let more through, and circuit breakers can disagree, until the last 3.1.0 pod is gone.
• Token limits start refusing. A tokens rate limit never refused a request before. It now does once its window is full, so a deployment with token limits will see 429s where it saw none.
• A key's limits no longer replace its owner's. A rate limit or budget set on an API key used to take the place of the owner's limit for the same window or period. Both now apply, each on its own counter: a key's limits can narrow what its owner may do through that key, never widen it. A key given a higher limit than its owner to give it more room needs the owner's limit raised instead.
• rediss:// connects over TLS. 3.1.0 connected to a rediss:// URL over plain TCP. A rediss:// URL pointing at a port without TLS now fails at start: point it at the TLS port, or write redis://.
  • The certificate must name the host in a subjectAltName. One that names it only in its CN, which redis-cli accepts, is refused.
  • Cluster nodes that announce IP addresses their certificates don't name can't be reached.
• budget_unavailable in dashboards. With Redis down and security.rate_limit_fail_closed on, a request that has a budget is refused as budget_unavailable, not rate_limiter_unavailable: budgets are checked first.
• Upgrade with an ordinary rollout. Earlier versions don't take the lock that now makes instances set up the schema one at a time, so don't restart instances of the old version while the first one of this version starts.
• No transaction-mode pooler in front of Postgres. Schema setup now holds a Postgres session-level advisory lock, which a pooler in transaction mode (PgBouncer pool_mode = transaction) can leave held, and every later start then waits for it: point DATABASE_URL at Postgres itself or at a pooler in session mode (the Helm chart's README has the details).

Fixed

• Token limits refuse requests. A tokens rate limit never refused anything, and stopped counting once a request would have taken it past its limit. A request is now refused once the window's recorded usage reaches the limit, and every request's tokens are recorded after it, even past the limit — a window can overshoot by what was in flight when it filled.
• Several request limits at once. With two or more requests limits on a user (per minute and per hour, say), every request that passed was counted twice, and only one of the limits could refuse; with security.rate_limit_fail_closed on, every request was refused as rate_limiter_unavailable.
• An API key's limits count on that key. They were counted on its owner's counter, which every key of the owner shared, and the usage the console reads for a key (/api/admin/limits/api_key/{id}/usage) was always 0. Each key now has counters of its own, rate limits and budgets alike, for the gateway and the MCP gateway, and its usage shows what it used.
• A refused request counts against nothing. A request refused by a spent budget, or by one rate limit after another had passed, was still counted against the request limits. Budgets are now checked first and every rate limit in one step, so a refused request leaves every counter as it was.
• Retry-After says when to retry. A 429 from the gateway's own limits said Retry-After: 30 whatever the limit. It now gives the seconds until the window has room for another request, or until a spent budget's period ends (the next midnight, Monday or 1st of the month, UTC). A spent budget also sends x-should-retry: false, so the OpenAI and Anthropic SDKs don't retry it by themselves. The body stays in the caller's API format.
• Redis Cluster. The rate-limit, route-health and quota scripts touched keys of several hash slots, which a Redis Cluster refuses: on a cluster, rate limits silently stopped applying (or refused every request with security.rate_limit_fail_closed), circuit breakers never tripped, and cache invalidation reached one node only. Every key a script touches now shares a hash tag, pattern deletes scan every node, and the Helm chart's README describes a redis-cluster:// URL.
• Redis over TLS. The server was built without TLS for Redis: a rediss:// URL was used as plain TCP, so against a Redis that requires TLS — ElastiCache with in-transit encryption, Upstash, Azure Cache for Redis and Redis Cloud among them — the server did not start (Failed to connect to Redis: Protocol Error: Expected string.). rediss:// and rediss-cluster:// URLs now connect over TLS and check the certificate against the system's CAs, as upstream HTTPS does. For a Redis whose certificate a private CA signed, REDIS_CA_CERT names a PEM file with that CA, which is then trusted alone; the Helm chart sets it from a Secret given in redis.caSecret. The chart's README describes both.
• Several instances starting at once. Server instances starting together against one database — a Helm replicaCount above 1, a rolling upgrade, an autoscaler adding pods — applied the schema side by side, and all but one could exit with Database migration failed: apply db/schema.sql: … deadlock detected (on an empty database: duplicate key value violates unique constraint "pg_extension_name_index"). With ClickHouse, the rollups that an instance fills from the logs when it finds them empty (cost_rollup_hourly, provider_health_5m, mcp_server_call_counts) could be filled by each of them, counting every request once per instance on the cost pages, the dashboard and the MCP server list. Instances now set up Postgres and ClickHouse one at a time, under Postgres advisory locks: the others wait, logging Another instance is setting up the database schema; waiting for it to finish, then find it done. An instance that dies holding a lock releases it with its connection. The Helm chart's startup probe allows a pod 125 s to start instead of 35 s, set in startupProbe (the chart's README says when to raise it), and an attempt to reach a ClickHouse that doesn't answer gives up after 5 s instead of the system's TCP timeout.
• Captured bodies kept as long as configured. With ClickHouse and audit.body_retention_days above 30, every server start could clear the captured request and response bodies older than 30 days (gateway_logs.request_body / response_body, mcp_logs.tool_arguments / tool_result; the rows themselves stayed). The start-up table setup set those columns' TTL to 30 days each time, and ClickHouse applies a TTL to the data already stored as soon as it is set, before the server put the configured TTL back a moment later. The setup now gives these columns a TTL only when it creates them, so a restart leaves the configured one in place. Bodies already cleared cannot be recovered. The log tables' own TTLs (data.retention_days_*) were not affected.
• Helm network policy and databases on other ports. With networkPolicy.enabled, the server could reach PostgreSQL only on 5432, Redis on 6379 and ClickHouse on 8123, whatever their externalUrl said, so a database on another port was blocked — Azure Cache for Redis over TLS (6380), a managed Postgres or a ClickHouse on a port of its own: the server could not start, or started without writing to ClickHouse. The allowed ports now follow postgres.externalUrl, redis.externalUrl and clickhouse.externalUrl: every port a URL names, and the client's default for its scheme where it names none. networkPolicy.extraEgress adds egress rules as written, for ports no URL names (Redis Cluster nodes announcing other ports, an upstream or MCP server on a port other than 443). The chart's README describes both. Port 9000, ClickHouse's native protocol, is no longer allowed: the server reaches ClickHouse over plain HTTP only (an http:// URL; HTTPS is not supported). An S3 endpoint on 9000 (RustFS, MinIO) configured outside the chart needs a rule in networkPolicy.extraEgress.

3.1.0 — 2026-10-05

The thinkwatch-core crates move from v0.59.0 to v0.62.0. Two changes reach the gateway: tool-call inspection gains a built-in rule for ThinkWatch's own data directory, and an upstream whose base URL already ends in an API version is no longer sent a second one.

Read before upgrading

• The new tool-call rule thinkwatch-data is on and set to cut off, like the other high-risk built-in rules. A deployment whose tool-call inspection is in enforce mode starts cutting off answers whose tool call reads or changes one of the paths below; one in observe mode only records them. Set the rule to record, or switch it off, on the security page to keep the previous behaviour.

Added

• Tool-call inspection: thinkwatch-data (Read or change ThinkWatch's own data). A tool call whose path or command points into ThinkWatch's data directory (~/.thinkwatch, %APPDATA%\ThinkWatch, /var/lib/thinkwatch, /etc/thinkwatch) — where the desktop gateway keeps every upstream key and its own protection settings. Text that only mentions the directory, such as a document being edited, does not count. The security page lists it with the other built-in rules.

Fixed

• Upstream base URLs that end in their own API version (…/api/paas/v4, …/api/v3) now receive requests under that version instead of a second /v1 appended to it, which returned 404. Requests and connection tests both use it.

Changed

• thinkwatch-core crates (tw-bedrock, tw-breaker, tw-dialect, tw-guard) v0.59.0 → v0.62.0.

3.0.0 — 2026-10-03

The request guards — outbound redaction, tool-call inspection and the content filter — now share their rule model with the desktop gateway: the same policy shape, built-in rule catalog, validation, rule view and sample trial, from thinkwatch-core. Each guard has three modes (off, observe, and one named for what it does: replace, cut off, enforce), and the content filter can delete what a rule matches as well as refuse or record it, and match by code point. Hidden characters become content filter rules, and the per-model output length guardrail becomes a cap on the output tokens a request may ask for. Settings saved by an earlier version are converted at the first start: read the first section before deploying. It is a major release because setting keys, console API routes and the models table change; the conversion keeps each deployment's behaviour.

Read before upgrading

• Back up system_settings and models first, and plan for no way back to 2.2. The upgrade converts the guard settings and drops a column (next item). Version 2.2 cannot run on the converted database: its settings are gone, so it would start on its seeded defaults, and it can no longer read the models table. Take a copy before deploying:

  pg_dump --data-only --table=system_settings --table=models "$DATABASE_URL" > thinkwatch-2.2-guards.sql

• Stop every replica of 2.2 before the first 3.0 one starts. A 2.2 replica still running when 3.0 converts finds its settings gone (it then filters and redacts nothing) and cannot rebuild its router; one that restarts writes its seeded defaults back. 3.0 does not convert those a second time — it removes them at its next start and logs a warning — but the 2.2 replica runs on them until then. With the Helm chart (release thinkwatch, namespace thinkwatch here):

  kubectl -n thinkwatch scale deployment/thinkwatch-server --replicas=0
  kubectl -n thinkwatch wait --for=delete pod \
    -l app.kubernetes.io/name=think-watch,app.kubernetes.io/component=server --timeout=5m
  helm upgrade thinkwatch deploy/helm/think-watch -n thinkwatch   # with your usual values
  kubectl -n thinkwatch scale deployment/thinkwatch-server --replicas=<replicas you run>

  The last command matters when autoscaling.enabled is on: the chart then leaves the replica count alone, and it stays at 0. With Docker Compose, docker compose stop server before pulling and starting the new image.

• The old guard settings are converted at the first start, and behave as before. security.content_filter_patterns, security.hidden_text, security.pii_redactor_patterns and security.tool_inspection become security.content, security.redact and security.inspect_tools, and are deleted, in one transaction during the boot migration. The conversion is recorded in security.legacy_converted (when, which version, what it converted); with that record present, old keys or the old column that show up again are removed, never converted, so the policies in force are not overwritten. A content rule identical to a built-in rule becomes that rule, switched on, any other a custom rule; a list with rules in it runs in enforce mode with every built-in rule it did not name switched off. The four seeded PII patterns become the built-in rules for the same data (cn-resident-id, bank-card, email, cn-mobile-phone), any other pattern a custom rule whose label is its old placeholder prefix. Whatever the old runtime was skipping (a rule that did not compile, a built-in rule id it did not know, an output_guardrails value it could not read) is left out, each with a warning in the start-up log. A model's output_guardrails length cap becomes max_output_tokens (below), and the column is dropped. To see what was converted, read the three keys from Settings or system_settings afterwards; the start-up log lists them too.

• Placeholders are written <<TW_EMAIL_1>>, not {{EMAIL_1}}. The label of a custom rule is upper case letters, digits and underscores (an old prefix is converted: REDACTED-SSN → REDACTED_SSN); the built-in identity number and bank card rules use ID_NUMBER and CARD_NUMBER. Anything that looked for the old form in answers or logs needs the new one.

• Redaction searches the whole request, not only the user's messages: the system prompt, earlier answers and tool-call arguments are redacted too. Base64 payloads (images, files, signatures) are still left alone. Rules run on the request as it is sent, as JSON, where a custom pattern's match ends at a quote or a backslash: a pattern written to match across a " in the decoded text needs rewriting.

• Built-in credential rules start replacing on deployments that were redacting. API keys and tokens with a known prefix, private keys, JWTs and connection-string passwords are built-in rules that ship switched on. A deployment whose PII list had patterns in it runs redaction in enforce mode after the upgrade, so these values are now replaced as well. Switch the ones you do not want off on the console's security page. With an empty PII list, redaction converts to observe mode: it records what it finds and changes nothing.

• The built-in rules that replace the seeded PII patterns are stricter. An identity number has to have a real province code, a real date of birth and a matching check digit; a card number a known network's prefix and length and a valid Luhn digit, and published test card numbers do not count. Numbers the old regexes took for them — any 18 digits, any 16 — are no longer replaced. A deployment that relied on the looser match can add its old regex back as a custom rule.

• Tool calls are judged as the client receives them, and two built-in rules are new. Inspection now reads a tool call converted to the caller's format and with redacted values restored — what the client would run — where it used to read the placeholders. The new secret-to-unknown-host rule cuts (in enforce mode) a call that sends a recognised API key or private key to a host that is neither local nor the key's own provider; upload-file-to-host records a call that uploads a local file to an outside host. A deployment running tool-call inspection in enforce mode starts cutting the first; add it to disable if that is not wanted.

• "Warn" and "log" are one action now, "record only", and hidden characters are content filter rules: unicode-tags and bidi-controls, plus zero-width and private-use, which ship off. security.hidden_text: block converts to those two rules refusing, warn and log to recording, off to switching them off. Recording writes an audit event: hidden_text: log used to reach only the application log, and now writes gateway.content_flagged.

• A deployment with no content rules starts recording. An empty content filter list converts to observe mode with the built-in rules that ship on, so requests matching them (ignore previous instructions and the like) write gateway.content_flagged events where 2.2 wrote nothing. Nothing on the wire changes.

• The output length guardrail is replaced by a model's maximum output tokens — check each model's after upgrading. A cap of N bytes on the answer converts to ceil(N / 4) output tokens, stored as converted. The answer is no longer measured or cut: a request asking for more tokens than the cap is lowered to it, in whichever field its API uses (a Chat request that sets both max_tokens and max_completion_tokens has both lowered), and the upstream stops there; one asking for less keeps its own. A request that sets no limit is held to the cap only when the cap is within what the gateway knows the model's family to take (32,000 tokens for Claude models, 8,192 for others); a Chat request is then given max_completion_tokens on OpenAI's own endpoint, whose reasoning models refuse max_tokens, and max_tokens elsewhere. Above that figure, the request goes out without a limit rather than with one the model could refuse, and the model's own default applies: a 100,000-byte cap on a non-Claude model converts to 25,000 tokens, so its requests that set no limit are not capped at all. To hold every request to a cap, have the clients send a limit, or set the cap to that family figure or below. Reasoning (thinking) tokens count towards the cap on the APIs that bill them as output, so a cap that fit an answer can cut short a model that thinks first. An Anthropic request with extended thinking has its thinking budget lowered below the cap too, or thinking turned off when the cap is 1,024 tokens or less, the smallest budget Anthropic takes. The model API's output_guardrails field is gone: a request that still sets one (anything but null or []) is refused with 400, so a script cannot believe answers are still capped. max_output_tokens (1 to 2147483647, null for no limit) replaces it.

• A new installation observes by default. Every guard starts in observe mode, with only the built-in rules that rarely misfire switched on (personal data such as e-mail addresses and phone numbers ships off). Nothing is refused, replaced or deleted until a guard is switched to its third mode.

• A content filter refusal is 403, with the error type of the caller's API (permission_error for OpenAI-style APIs). Keyword and regex rules used to refuse with 400. In gateway_logs, its error_type is PolicyBlocked, where it was TransformError.

• A rule that deletes text changes what is stored. The request a content rule stripped goes upstream, is redacted and is captured as the stripped one: the audit log's request body is what was sent, not what the caller typed.

• Changing a guard policy takes settings:write and the guard's own permission: pii_redactor:write for security.redact, content_filter:write for security.content and security.inspect_tools, through PATCH /api/admin/settings. 2.2 checked settings:write on the server and the guard permission in the console; both are checked on the server now. Trying a sample takes pii_redactor:read or content_filter:read. Reading the policies — GET /api/admin/security, which the console's security page loads — takes settings:read. The seeded admin and super_admin roles hold all of them.

• Console API changes. GET /api/admin/security lists each guard's mode and every rule, and POST /api/admin/security/{guard}/test tries a sample; they replace /api/admin/settings/content-filter/test, /content-filter/presets, /pii-redactor/test, /tool-inspection/rules and /tool-inspection/test, which are gone.

• Audit events. Every rule that matches writes one event per request: gateway.content_flagged, gateway.content_stripped and gateway.content_blocked; gateway.redaction_flagged and gateway.redaction_replaced; gateway.tool_call_flagged and gateway.tool_call_blocked as before. gateway.hidden_text_flagged and gateway.hidden_text_blocked are gone; hidden characters are content events. No event carries the request's text: a content event names the rule, the outcome, how many matches and whether they were in a tool result; a redaction event names the rule and counts the values and their occurrences, and only a built-in rule's lists a few in masked form (sk-an…7f9c) — a custom rule's values are not written at all. A request writes at most 20 events per guard (the rules that changed it, or matched most, first), each with the number of rules that matched (rules_in_request). With audit.body_redact_pii on, captured bodies are redacted with the outbound redaction rules, built-in ones included, whatever the redaction mode.

• Metrics renamed. gateway_hidden_text_total is gone: hidden characters count in gateway_content_matched_total{outcome,custom} with every content rule. pii_pattern_invalid_total{pattern} is now guard_policy_invalid_total{guard} (a stored policy with a rule that does not compile; the rule is left out), next to guard_policy_unreadable_total{guard} (a stored policy that cannot be read; the guard runs on its factory policy). Redaction counts values in gateway_redaction_found_total{kind,outcome}.

Added

• Deleting what a content rule matches. A content rule can refuse the request, delete the matched text from the caller's messages and tool results and send the rest, or only record. Text deleted joins back what it separated, so the request is checked again afterwards.
• Code point rules. A content rule can match characters by code point (U+200B, U+E0000–U+E007F), for invisible characters a keyword cannot be written for.
• Every rule visible and switchable, built-in and custom, in each guard, with what it does in the third mode and what it did out of the box; a sample can be tried against one rule, an unsaved one, or all of them.

Changed

• Core crates at ThinkWatch-Core v0.59.0. tw-dialect, tw-guard, tw-breaker and tw-bedrock move from v0.55.0; the shared guard model described above comes with them.

Fixed

• A credential in a matched tool call no longer reaches the audit log. The excerpt of a tool call that inspection cut or recorded is masked with the redaction rules before it is written; 2.2 stored the matched arguments as they were, a key the model wrote in them included.
• A request's audit events and its log row carry the same id when the caller sends no x-trace-id. The log row of a request that went through used to carry a second, unrelated id.

2.2.0 — 2026-10-01

This release fixes authorization. The gateways never checked ai_gateway:use or mcp_gateway:use, so a user with no role, or with only roles that grant neither, could call every model and MCP tool; a role granted at the scope of one team widened model and tool access platform-wide; and an API key whose allow-list came out empty could call any model. The gateways now require the permission and count only the roles that grant it. The seeded team_manager role now works when granted at team scope, and seven permissions that nothing ever checked are retired. Some users lose gateway access on upgrade: read the first section before deploying. There is one manual database script and no schema, setting, environment variable or Helm value change.

Read before upgrading

• Users without a role that grants gateway use lose gateway access. A request to the AI gateway is refused with 403 unless a role held by the key's owner grants ai_gateway:use, and a request to the MCP gateway unless one grants mcp_gateway:use. The body names the missing permission, in the error format of the caller's API. The built-in developer, admin, super_admin and team_manager roles grant both; viewer grants neither. Three groups of users are refused after the upgrade where 2.1.0 let them through:

  • users with no role at all;
  • users whose roles are only viewer, or custom roles without these actions;
  • users whose only gateway role is assigned at team scope (see the third item below).

  Find them before upgrading. This query lists every active user with a live key for a gateway that none of their roles will grant (it counts global assignments and roles attached to the user's teams, which is what the gateways now read; it does not account for Deny statements):

  WITH grants AS (
    SELECT id AS role_id,
           jsonb_path_exists(policy_document,
             '$.Statement[*] ? (@.Effect == "Allow").Action[*] ? (@ == "*" || @ == "ai_gateway:*" || @ == "ai_gateway:use")') AS ai,
           jsonb_path_exists(policy_document,
             '$.Statement[*] ? (@.Effect == "Allow").Action[*] ? (@ == "*" || @ == "mcp_gateway:*" || @ == "mcp_gateway:use")') AS mcp
      FROM rbac_roles
  ),
  held AS (
    SELECT user_id, role_id FROM rbac_role_assignments WHERE scope_kind = 'global'
    UNION
    SELECT tm.user_id, tra.role_id
      FROM team_members tm JOIN team_role_assignments tra USING (team_id)
  ),
  access AS (
    SELECT h.user_id, bool_or(g.ai) AS ai, bool_or(g.mcp) AS mcp
      FROM held h JOIN grants g USING (role_id) GROUP BY h.user_id
  )
  SELECT u.email, s.surface
    FROM api_keys k
    JOIN users u ON u.id = k.user_id
    CROSS JOIN LATERAL unnest(k.surfaces) AS s(surface)
    LEFT JOIN access a ON a.user_id = u.id
   WHERE k.is_active AND k.deleted_at IS NULL
     AND u.is_active AND u.deleted_at IS NULL
     AND NOT COALESCE(CASE s.surface WHEN 'ai_gateway' THEN a.ai
                                     WHEN 'mcp_gateway' THEN a.mcp END, false)
   GROUP BY u.email, s.surface
   ORDER BY u.email, s.surface;

  Give each of them developer (or a custom role with the actions) either at global scope (Users → edit the user's roles, scope Global), or by attaching the role to a team they belong to (Teams → the team → Roles), which every member of that team inherits. The change takes effect within a minute; each user's permissions are cached for 60 seconds.

  If SSO users are meant to use the gateway from their first sign-in, set Default Role for New Users (auth.default_role) in Settings to developer. It is empty by default, and it applies only to accounts created after it is set; existing users need the grant above.

• A role that does not grant gateway use no longer widens model or tool access. Model and MCP tool scopes are now the union over the roles that grant ai_gateway:use / mcp_gateway:use only. A role without those actions (such as viewer) used to count as "unrestricted", so adding it to a user limited to some models opened every model to them. Users relying on that lose the extra models.

• A role assigned at team scope no longer grants gateway access. An assignment with scope team:<id> administers that team from the console (team roster, team limits); it no longer contributes models, MCP tools or gateway use, which it used to do for every request the user made, member of the team or not. Roles attached to a team itself (Teams → Roles), which every member inherits, still count. A user whose only gateway role was a team-scoped developer or team_manager needs that role at global scope, or attached to their team. The role assignment editor now says this under the scope picker.

• An empty model allow-list allows nothing. A key whose allowed_models is [], or whose list shares no model with what its owner's roles grant, used to call any model; it now calls none, as allowed_mcp_tools: [] already did on the MCP gateway. The console never saves [] (clearing the picker sends null), so only keys written through the API are affected. To find them: SELECT id, name, user_id FROM api_keys WHERE allowed_models = '{}' AND is_active AND deleted_at IS NULL; Set such a key's list to null to leave it bounded by its owner's roles only. Model entries still match by prefix, and a key narrowed to gpt-4o-mini under a role granting gpt-4o keeps gpt-4o-mini.

• Run db/release_migrations/2026-09-30_retire_unchecked_permissions.sql once, after deploying. The server does not apply it. It adds teams:read to the team_manager role and removes the seven retired permissions (see Changed) from every role, system and custom. It is idempotent and changes no access for any other role, since nothing checked the removed keys. Without it, the server still starts and works, but:

  • team_manager keeps the old team:read it cannot use, so a team manager who is not a member of the team they manage still gets 403 opening it, its roster or its roles;
  • every start logs a warning listing the roles that still name retired permissions.

  Reset to defaults on a system role in the console has the same effect for that one role, but does not clean custom roles.

• A team-scoped rate_limits:write now takes effect. It used to require global scope for every subject, so the seeded team_manager granted at team scope could not touch any limit. It now covers the rate limits and budget caps of users in that team and of the API keys they own, never the holder's own user or keys; role subjects still need global scope. Anyone holding team_manager (or a custom role with rate_limits:write) at team scope can now change, lift or delete the limits of every member of that team, administrators included. Review team-scoped assignments of these roles before upgrading.

Security

• Gateway use is checked, and a missing grant means no access. See the first three items above: a user with no role, or only roles without gateway use, could call every model and MCP tool; a viewer-style role widened a restricted user to every model; a role granted at the scope of any team, even one the user was not a member of, widened model and tool access platform-wide. An explicit Deny on ai_gateway:use or mcp_gateway:use now closes that gateway. Action wildcards (ai_gateway:*, *) grant it, as they already did for console permissions.
• A key narrowed inside a prefix grant is no longer unrestricted. The key's allow-list was intersected with its owner's role grants entry by entry, as literal strings. A key limited to gpt-4o-mini under a role granting gpt-4o (a prefix) came out with an empty list, which the gateway read as "no restriction", so the key could call every model. The intersection now keeps an entry of either side that the other side covers, by prefix for models and by <server>__* pattern for MCP tools, and an empty result allows nothing.

Fixed

• The seeded team_manager role works at team scope. It granted team:read, but the team handlers check teams:read, so a team manager got 403 listing teams and opening the team, its roster or its roles. It now grants teams:read, and opening a team accepts teams:read scoped to that team, where it used to require global scope. Existing installations need the release migration above.
• Bulk disable and delete work on API keys' limits. The bulk disable and delete routes for rate-limit rules and budget caps rejected every row stored for an API key (api_key_lineage, the kind a key's limits are stored under) as an unknown subject kind.
• The console's effective-permissions preview shows real gateway access. It now counts only roles that grant gateway use and skips team-scoped assignments, as the gateways do, where it used to count every assigned role.

Changed

• Seven permissions that nothing checked are retired: team:read, team:write, logs:read_own, logs:read_team, audit_logs:read_own, audit_logs:read_team and audit_logs:read_all. Every log endpoint, audit logs included, is gated on logs:read_all at global scope, and no own- or team-filtered log view exists, so these grants never did anything. They are gone from the permission catalog, the role editor and the seeded roles. Roles that still name them load, and the server logs a warning at start instead of refusing to boot.
• Deleting one rate-limit rule or budget cap is bound to the subject in the path. DELETE on /api/admin/limits/{kind}/{id}/rules/{rule_id} and /api/admin/limits/{kind}/{id}/budgets/{cap_id} now answers 404 when the rule or cap belongs to a different subject; it used to delete any row id once the path's subject was authorized.
• Creating or editing a key with an explicit allow-list checks it against gateway-granting roles only. An allowed_models or allowed_mcp_tools entry is refused with 400 when no role of the owner that grants the gateway covers it, so a user without a gateway role can no longer save a non-empty list.

2.1.0 — 2026-09-30

Amazon Bedrock becomes a provider you can run from the console. It authenticates with a Bedrock API key as well as access keys or the instance role, lists its models for import and for the route editor, and has a working Test Connection. Requests converted for Bedrock now keep their prompt-cache breakpoints and Claude's thinking settings. The route editor takes any upstream model name, and a route the provider refused can be created anyway. ThinkWatch-Core moves to v0.55.0, whose Bedrock layer this edition now shares with the desktop gateway. No database, setting, environment variable or Helm value changes.

Read before upgrading

• Test Connection with a saved provider's secrets needs providers:update. A test that names a saved provider (provider_id, which is what the Edit dialog sends) used to take only providers:create. A custom role that has providers:create without providers:update can no longer test existing providers; the built-in admin role has both. A test with values typed into the request still takes providers:create. See Security below.
• Refusing a route the provider does not serve answers model_not_served. POST /api/admin/models/{model_id}/routes still answers 400 when the import probe found no API the provider serves the model on, but error.type is now model_not_served, where it was bad_request. Scripts that matched on bad_request for this case need the new value. The request takes a new optional "force": true to create the route anyway.
• error_type in gateway_logs for failed requests is always the error's tag. An upstream HTTP error used to log its debug text, such as ProviderHttpError { status: 502, message: "…" }, and an upstream rate limit a cut-off UpstreamRateLimited { retry_after_secs: Some. They now log ProviderHttpError and UpstreamRateLimited, as the metric labels and streamed requests already did. Dashboards or log forwarder queries that grouped by those strings see them merge into one value each.
• The server log now carries the upstream's reply to a 401 or 403, as it already did for other upstream errors. For Bedrock that reply names the AWS account and the IAM principal. Callers still see only "Authentication failed with upstream", and failover, breakers and gateway_logs treat these errors as before.
• Prompt caching now works on Bedrock routes, and is billed at cache prices. Requests converted for Bedrock used to lose their cache_control breakpoints, so Claude on Bedrock re-read the whole prompt at full price every turn. Breakpoints now go out as Converse cachePoint blocks, to Claude and Nova models only (others reject them), and the cache reads and writes Bedrock reports come back into usage, one-hour writes included. Traffic with repeated prompts, such as Claude Code, costs much less on Bedrock routes than it did; writes cost a little more, at cache_write_weight / cache_write_1h_weight.
• Claude's thinking reaches Bedrock. A request converted for Claude on Bedrock now carries its thinking and effort settings, in the form Anthropic's API uses, along with the sampling limits thinking imposes (temperature 1, no top_k, top_p at least 0.95). Thinking used to be dropped on the way to Bedrock. Other Bedrock models still get no thinking.

Added

• Bedrock providers can authenticate with a Bedrock API key. Pick Bedrock API Key as the authentication mode and paste the key AWS generated. It is kept like any other provider's API key, as an encrypted Authorization: Bearer header, and replaced in the provider's Edit dialog. A Bedrock provider that sends an Authorization header is not SigV4-signed; one without it is signed as before, with access keys or the instance role. Use a long-term key: a short-term one expires within 12 hours.
• Bedrock models can be imported, and are suggested in the route editor. A Bedrock provider now lists what it can be routed to: the region's foundation models that can be invoked on demand and answer in text, and the inference profiles AWS defines, such as us.anthropic.claude-sonnet-4-5-20250929-v1:0. A model that is only served through an inference profile, as most current models are, is listed under its profiles' ids and not its own. Whether the account may call a model is still checked one model at a time when it is imported. The provider's credential needs bedrock:ListFoundationModels and bedrock:ListInferenceProfiles; the AmazonBedrockLimitedAccess policy a long-term API key is created with allows both.
• Test Connection for Bedrock providers, however they authenticate: with an API key, access keys or the instance role. It lists the models above, so a wrong credential or a missing permission shows up before any traffic does.
• Create anyway, for a route the provider refused. When the route editor's save is refused because the provider does not serve the model, the error now offers Create anyway: the refusal can be stale, or be about the probe's request rather than the model. The route is created, and its model_route.created audit row records the refusal it overrode in a new refusal_overridden key.

Changed

• The route editor's upstream model field takes any model name. For a provider that lists its models, the field used to turn into a list to pick from, so a model the listing leaves out could not be routed to from the console. It now suggests the provider's models as you type, and takes whatever is typed.
• Bedrock instance-role credentials are cached. They took three IMDSv2 round trips per request; they are now kept until five minutes before they expire, and a burst of requests at expiry makes one trip.
• Chat Completions requests can switch reasoning with thinking. When a request is converted for another format, thinking.type (enabled / disabled, as DeepSeek, GLM and Kimi write it) now turns reasoning on or off; disabled wins over reasoning_effort.
• Core crates at ThinkWatch-Core v0.55.0. tw-dialect, tw-guard and tw-breaker move from v0.43.0, and tw-bedrock joins them: SigV4 signing, eventstream unframing, Bedrock's addresses, region checks and model catalog now come from core. Where credentials come from (provider keys, the instance role) stays in this repository. tw-breaker is unchanged.

Fixed

• Bedrock providers could not be created or edited in the console. The region was checked as a URL, so saving failed with 400 Invalid URL. A Bedrock provider's base_url is now checked as an AWS region such as us-east-1, and anything else is refused, since the host is built from it. A provider saved with a URL there never reached Bedrock; set its region in the Edit dialog.
• Bedrock models the account may not call were imported anyway. Bedrock refuses such a model (model access not granted, or an IAM or organization policy that denies it) with a 403, and the import probe read every 403 as a credential problem that says nothing about the model. The route was created and failed on first use. Now, when the region's control plane accepts the same credential, the refusal is recorded as the model's, with AWS's reason, and the model is skipped on import like any other refused one. Once access is granted, re-check the provider's models.
• A Bedrock provider whose stored secret key would not decrypt signed with an empty secret, and every request failed with AWS's SignatureDoesNotMatch. It now refuses its requests with the reason until the keys are saved again, rather than falling back to the instance role, which would call AWS as a different identity. An access key ID saved without a secret is refused the same way.
• A Bedrock model id that is an ARN could not be routed. Its / went into the request path unescaped, adding a path segment AWS could not route. It is now escaped.
• A base URL ending in its version segment doubled it. A provider written as https://api.openai.com/v1 sent requests, and the model listing, to /v1/v1/… and got 404s. A trailing version segment (v1, v1beta, …) that the request path starts with is now written once. Base URLs without one are unchanged.
• Typing into a provider's API key field and clearing it again wiped the saved key on save. The field sent Bearer with nothing after it. A cleared field now keeps the saved key, as a blank one always did.

Security

• Testing a connection with a saved provider's secrets takes providers:update. The test fills each header left blank with the saved provider's secret and sends it to the URL in the request, and providers:create was enough to ask for it. So a user allowed only to create providers could send any saved API key to a server of their own. A test that names a saved provider (provider_id) now takes providers:update, which lets a user point that provider elsewhere anyway; a test without one still takes providers:create. A user with providers:update alone can now use Test Connection in the Edit dialog, which used to answer 403.

2.0.0 — 2026-09-24

Callers now get errors in their own API's format, and an upstream that refuses a request no longer takes a model's other routes down with it. The gateway also speaks two more client protocols: Gemini, and the Responses API over a WebSocket. Cached input is billed at cache prices, and a request with no usage report is billed on an estimate instead of at zero. The TOTP requirement, which never took effect before, is now enforced. This is a major release because error bodies, the content-filter preset ids and the TOTP behaviour all change in ways a client or a script can notice.

Read before upgrading

• Check security.totp_required before you upgrade. In 1.x this setting never took effect: it is stored as a boolean and was read as a string, so it always read as off. From 2.0.0 it is enforced by the server. Find out what it is set to:

  SELECT value FROM system_settings WHERE key = 'security.totp_required';

  If it is true, every console user without TOTP, super admins included, is held at a TOTP setup screen on their next request, and sessions that are already open are held too. Until they set up TOTP, every console and admin endpoint answers 403 totp_enrollment_required, except /api/auth/me, logout, register-key and the TOTP status/setup/verify-setup calls. Setting up TOTP releases the session straight away. API keys are not affected: gateway, MCP and console tw- key traffic keeps working. While the setting is on, POST /api/auth/totp/disable is refused with 400. The setting must now be a JSON boolean; a string such as "true" is refused on save.

• Gateway error bodies follow each client API's own format. Status codes and Retry-After are unchanged. Code that reads the error type needs updating:

  • Chat Completions and Responses keep the {"error": {"message", "type", …}} shape, but type is now OpenAI's value for the status, not a ThinkWatch tag: authentication_error (401), permission_error (403), not_found_error (404), rate_limit_error (429), invalid_request_error (other 4xx) and server_error (5xx). The old tags are gone: rate_limited, policy_blocked, provider_http_error, provider_error, provider_timeout, transform_error, network_error and auth_error. A policy block is now permission_error with status 403.
  • Anthropic Messages clients get Anthropic's body, {"type": "error", "error": {"type", "message"}}, with Anthropic's type names (rate_limit_error, overloaded_error, api_error, …).
  • Gemini clients get Google's body, {"error": {"code", "message", "status"}}.
  • Once a stream has started, a Responses client gets a response.failed event, where before it got a Chat-style error frame that SDKs skip, so the stream just stopped. An Anthropic client gets an error event whose type follows the status.
  • The error_type field in gateway_logs and the metric labels are unchanged.

• Only upstream failures fail over or count against a route's breaker. In 1.x every non-2xx except 401, 403 and 429 was retried on the model's other routes and counted as a failure on each of them, so one malformed request could open the breakers on all of a model's routes.

  • Tried on another route and counted against this one: 5xx, 408, 429, 401 and 403 (the upstream refused the gateway's own credential), timeouts, network errors and unreadable responses.
  • Returned to the caller straight away, and counted as the upstream working: every other 4xx.
  • What the caller sees: such a 4xx comes back with its own status and the upstream's reason. In 1.x it came back as a 502 after every route had been tried.
  • An upstream 5xx comes back with the upstream's status (500, 503, …) rather than a blanket 502.
  • An upstream timeout is now 504.
  • Streams follow the same rule. A stream cut by tool-call inspection no longer counts against the route.

• Cached input is billed at cache prices. In 1.x cache reads and writes were billed, and debited from budgets and weighted rate limits, as full-price input. models gains three weights, cache_read_weight, cache_write_weight and cache_write_1h_weight. When a weight is unset, it is input_weight times Anthropic's ratio: 0.1× for a read, 1.25× for a write and 2× for a one-hour write. What this changes:

  • Traffic with many cache reads (Claude Code, for instance) costs much less than it did.
  • Traffic that writes to the cache costs a little more.
  • Older OpenAI models discount cache reads less (0.5× or 0.25×). Set the weights on those models yourself.
  • input_tokens in the log is still the whole input. The log detail gains cache_read_tokens, cache_write_tokens and cache_write_1h.

• Output length limits now apply to streams. In 1.x, max_length output guardrails checked only whole responses, so streamed answers were never checked. Now the frame that would cross the limit is not sent, and the stream ends with an error in the caller's format. A response served from the cache is also checked against the limit in force. If you set a limit, streamed answers that used to go through can now be cut off.

• Content-filter preset groups are renamed. The groups are now injection, persona and chinese; they used to be basic, strict and chinese. This matters only if you call the presets endpoint by group id. Rules you have already added are copies and are not affected. Other changes to the filter:

  • A rule with an empty pattern is now refused on save.
  • Each text part of a message is scanned separately, so a pattern no longer matches across two parts.
  • The engine is now shared with ThinkWatch-Core's tw-guard. The stored format and the admin API are unchanged.

• Requests with no usage report are billed on an estimate. In 1.x such a request was billed at zero. This happens when an upstream ignores the request for usage, or when the caller leaves before the final chunk arrives. The estimate is:

  • input: about four bytes of the request per token, not counting images and files;
  • output: the answer that actually arrived.

  Estimated rows carry usage_estimated: true in their detail and count in gateway_usage_estimated_total. A request with no answer at all is still billed at zero.

• Clients that leave early are now logged. In 1.x a client that disconnected before its response existed left no gateway_logs row at all. That covers leaving during auth, limits or routing, or while waiting for a whole (not streamed) answer. Such a request now writes one row: status 499, stream_outcome: client_cancelled, cancelled_before: response, no tokens and no cost. Expect more 499 rows in dashboards and log forwarders. A new counter, gateway_cancelled_before_response_total, counts them.

Database changes

Both apply on their own at startup, as every schema change does, and both are additive:

• models gains three nullable columns: cache_read_weight, cache_write_weight and cache_write_1h_weight (ALTER TABLE … ADD COLUMN IF NOT EXISTS, CHECK (>= 0)).
• system_settings gets an auth.default_role row, seeded empty (no role). Existing rows are left alone (ON CONFLICT DO NOTHING).

Neither is irreversible. A 1.1.0 server runs against the upgraded database: it ignores the new columns and the new setting. What a 1.1.0 server cannot do is price cache tokens from the weights.

Added

• Gemini clients. New endpoints:

  • POST /v1beta/models/{model}:generateContent and :streamGenerateContent, also served under /v1/models/…;
  • GET /v1beta/models, which lists models in Gemini's format.

  These requests get the same limits, budgets, filters, routing with failover, format conversion, inspection, billing and audit as every other endpoint. A Gemini upstream gets the request as it was sent. A stream comes back as SSE with alt=sse, and as Gemini's JSON array without it. :countTokens and :embedContent are refused with 400.

• The Responses API over a WebSocket. Connect to GET /v1/responses with Upgrade: websocket.

  • Each response.create frame is handled like a streamed POST /v1/responses, with its own limits, routing, billing and audit row.
  • Turns on one connection run in order. A refused turn fails with response.failed, and the connection stays open.
  • The connection keeps its latest response. That lets a turn continue from it with previous_response_id, even with store: false, which is how Codex works, and against any upstream format.
  • A new counter, gateway_responses_ws_connections_total, counts connections.

• More places to put an API key. Gateway keys are also accepted in x-api-key (Anthropic SDKs), x-goog-api-key and ?key= (Gemini SDKs), as well as Authorization: Bearer. Headers are checked first. A key given in the query string is never sent upstream.

• Hidden-text audit events show what the text says. Each item in found gains revealed, the ASCII that the hidden tag characters spell.

• auth.default_role can be set. It is the role that newly registered users and SSO users get. In 1.x, setting it through the admin API reported success but changed nothing, because the setting row did not exist.

• Model editor fields for the three cache weights. Each placeholder shows the value used when the field is left empty.

Changed

• Default output length for upstreams that require max_tokens. When the caller sets none, the gateway now sends 32000 for Claude models and 8192 for other models. It used to send 4096, which cut Claude answers short.
• More upstreams count as the vendor's own endpoint. DeepSeek, Moonshot, Zhipu/Z.ai, DashScope, xAI and *.amazonaws.com are now recognised, and the check reads the parsed host. A relay URL such as https://relay/api.openai.com no longer passes as official. Official endpoints are stricter about request parameters, so the gateway drops or renames some parameters before sending to them.
• Hidden-text scanning uses ThinkWatch-Core's tw-guard. Same scope, same actions, and nothing is stripped.
• Requests forwarded in their own format lose ThinkWatch's reasoning signatures. A tw1. signature written by an earlier format conversion is removed, because Anthropic rejects it. The upstream's own signatures are kept.
• Core crates: tw-dialect, tw-guard and tw-breaker at ThinkWatch-Core v0.43.0. The code only this edition used (at-rest crypto, SigV4 signing, the gateway error type) moved into this repository. It works the same, and stored secrets decrypt as before.
• The server's SQL moved from the request handlers into repository modules (catalog, dashboard, limits, log forwarding, identity, access, MCP). Every statement is unchanged. New integration tests cover these endpoints and pass on both the old and the new code.
• CI runs on pull requests into dev, including the whole integration suite against Postgres, Redis and ClickHouse.

Fixed

• Revoking a user's default MCP connection always failed with a 500, and the account could not be revoked. The newest remaining account is now made the default.

1.1.0 — 2026-09-24

The gateway stops rebuilding every request as a chat-shaped message. A request whose route speaks the caller's own format goes out as the caller sent it; one that crosses formats is converted by ThinkWatch-Core, the same layer the desktop edition uses. Tools, tool choice, system prompt blocks, metadata and cache_control now reach the upstream, where they used to be dropped. Two new checks guard what goes in and out: tool calls an upstream returns, and invisible characters in what a caller sends.

Read before upgrading

• Anthropic routes record more prompt tokens for the same work. Prompt tokens now count the same for every upstream: plain input plus cache reads and cache writes, which is OpenAI's definition. Anthropic's own input_tokens leaves the cached part out, so on those routes prompt tokens, cost and budget use all go up. The price model still charges every prompt token at one rate.
• Requests in the upstream's own format are forwarded as sent. Every field the caller sends reaches the upstream, along with the caller's anthropic-beta and anthropic-version headers. Only the model name changes, and PII is swapped for placeholders. An OpenAI-compatible upstream that rejects fields it does not know may now refuse requests that used to succeed, because those fields were stripped before. Try your upstreams with the clients you actually run.
• Two checks are on by default, and neither blocks anything yet.
  • Tool-call inspection starts in observe mode.
  • The hidden-character check starts in warn mode.
  • Both write audit events, so expect new entries in the audit log and in anything subscribed to it.
  • Nothing is refused until you switch to enforce or block.
• The response cache starts cold. The cache key now covers the whole request, so entries written by 1.0.2 are never hit again. They expire on their own.
• One PII value gets one placeholder. Within a request, the same e-mail address is {{EMAIL_1}} wherever it appears. It used to get a new number each time, so a model saw one person as several. Saving a PII pattern now also requires the placeholder prefix to be letters, digits or underscores.

Added

• Tool-call inspection (security.tool_inspection).
  • Why: an upstream writes the response, so it can hand the caller a tool call the model never made, such as bash("curl … | sh") appended to an ordinary answer. An agent set to auto-approve then runs it.
  • Rules: a built-in set of dangerous-command rules. Each can be switched off or given a different action, and you can add your own.
  • Modes: off, observe (records hits and changes nothing on the wire) and enforce.
  • Enforce on a stream: the stream is cut at the frame that would complete a matching call, and the refusal arrives in the caller's format.
  • Enforce on a whole response or a cache hit: refused with 403 (policy_blocked). A refused answer is neither cached nor billed.
  • Audit and metrics: every hit is an audit event (gateway.tool_call_flagged or gateway.tool_call_blocked) and counts in gateway_tool_call_flagged_total.
  • Admin API: GET /api/admin/settings/tool-inspection/rules and POST /api/admin/settings/tool-inspection/test.
  • Console: a card on the security page, plus a sandbox tab.
• Hidden-character check (security.hidden_text: off, log, warn or block).
  • What it looks for:
    • Unicode tag characters, which carry an instruction invisibly into the model's context;
    • bidirectional overrides, which make text read differently on screen than it is.
  • Where: the caller's messages and the tool results inside them. The system prompt and the model's own turns are not checked.
  • Not flagged: zero-width joiners (emoji), the zero-width non-joiner (Persian) and Cyrillic.
  • Actions: warn writes gateway.hidden_text_flagged; block refuses with 403 and writes gateway.hidden_text_blocked.
  • Console: a card on the security page.
• Tool-call arguments get their PII back. A model asked to e-mail a@example.com used to call the tool with {{EMAIL_1}} as the address.

Changed

• One pipeline for /v1/chat/completions, /v1/messages and /v1/responses. A same-format request is forwarded as sent. A cross-format request is converted, and the gateway logs what the target format cannot carry.
• Content filtering and PII detection read tool results too. That is where an injected instruction, or customer data pulled in by a tool, usually sits.
• Usage is read off the upstream's own bytes. A streamed response is no longer held in memory for an accounting pass at the end.
• Chat streams are billed on the upstream's actual usage. They are always sent asking for it. A caller who did not ask for the usage chunk still does not get one. 1.0.2 estimated the count for these.
• Streams send their headers at once. A caller who leaves while the upstream is still thinking is recorded as cancelled.
• Connectivity tests use the live encoder. A route's test request is built by the same encoder as real traffic, so a passing test means forwarding works.
• The web console loads data through TanStack Query.
  • Signing out, including from another tab, clears everything cached.
  • After a change, screens refresh in place.
  • Polling pauses while the tab is hidden.
• Core crates come from one pinned tag (ThinkWatch-Core v0.40.0), declared once at the workspace root.

Fixed

• Requests lost their tools, tool choice and non-text content on the way upstream (ThinkWatch-Core#50). Claude Code's system prompt, sent as an array, was dropped whole, and so was every cache_control breakpoint. Each cached prefix was billed as full-price input.
• The response cache could serve the wrong answer. Its key covered only model, messages and max_tokens, so two requests that differed only in tools, response_format, seed and so on shared one entry.
• A tripped route stayed out until its Redis key expired, roughly four cooldowns. It now gets probed once the cooldown is over, and a success closes it.
• The dashboard showed every AI provider's breaker as Closed. It now shows the real state of the provider's routes, reporting the worst one.
• The PII "try patterns" endpoint misreported labels. A pattern named CUSTOM_EMAIL was reported as CUSTOM.
• :latest could point at a main build rather than the release, which is what happened for v1.0.2's server image. Only the release workflow sets :latest now.
• The web image hung for six hours. Its frontend was built under QEMU for arm64, where Node crashed and the step never returned. It is now built once, natively. The image contents are unchanged.

Security

• Refreshed the web console's lockfile to clear 56 Dependabot alerts (1 critical, 23 high). All were transitive, and none of them reached the shipped bundle.

1.0.2 — 2026-09-13

The shared gateway layer moves out into its own repository, and OIDC learns to accept an ID token whose aud carries more than the client ID. Mostly a fix release otherwise — the deploy and auth items below are the ones worth reading before you upgrade.

Added

• OIDC_ADDITIONAL_TRUSTED_AUDIENCES — a comma-separated allowlist of extra ID-token audiences to trust. Some IdPs put something besides the client ID in aud; Zitadel includes the parent project ID, and openidconnect's stock verifier rejects every non-client-ID audience, so login failed outright. Leave it unset and verification is byte-for-byte what it was. Set, it widens exactly one check: the extra audiences on a multi-audience token are matched against this list as exact strings — no wildcards, no prefixes. The client ID must still appear in aud regardless. Documented in .env.example. Thanks to @DaniW42 for the report and the patch (#12).
• Automatic upstream protocol detection — the gateway learns an upstream's dialect and relearns it when the upstream changes, instead of relying on a static guess. Protocol is now a property of the route.
• Reverse-proxy identification — the server recognizes its own reverse proxy, which is what makes the dashboard WebSocket and the API docs work behind nginx.

Changed

• azp is validated whenever it is present, per OIDC Core 3.1.3.7 step 5, which has no audience-count precondition. Previously it was only checked when a multi-audience token made it mandatory, so a single-audience token naming this client with azp pointing at a different client was accepted — the IdP stating plainly that the token was authorized for somebody else. This can reject a token that 1.0.1 accepted. If your IdP sets azp to something other than your client ID, logins will start failing and the message will say so; that is the IdP to fix, not this setting.
• The shared gateway layer now comes from ThinkWatch-Core as a git dependency rather than living in this tree. Six crates — types, protocol, provider, resilience, crypto, and the JSON secret envelope — are one implementation shared with the desktop edition instead of two copies drifting apart. No runtime or API change; it affects you only if you build from source, where the build now needs network access to resolve that dependency. The published images are unaffected.
• Stored secrets are redacted on read and decrypted on use. Provider credentials no longer travel in plaintext through code paths that merely display or list them.

Fixed

• Deploy — nginx broke the API docs three separate ways; the dashboard WebSocket wasn't proxied; the prod stack's healthchecks were wrong; and the dev stack could overwrite production's ClickHouse credentials. The last one is the reason to read this list.
• Auth — the console logged itself out after every token refresh. Rate-limit counters could be created without an expiry and sit in Redis forever.
• Models — the gateway no longer offers models the upstream refuses, no longer imports models it won't serve, and reports every dialect that was refused rather than only the last. Provider base_url trailing slashes are normalized.
• Analytics — spend is attributed to a person, not a UUID.
• Audit — a syslog forwarder formatting fix that had CI red on clippy 1.98 (#11, thanks @DaniW42). Two more instances of the same lint, plus a result_large_err false positive in the MCP lifecycle stages, measured rather than boxed: the Ok variant is 296 bytes against the Err's 152, so boxing saves nothing and adds an allocation.
• UI — the brand mark stays inside the collapsed sidebar rail.

Contributing

CONTRIBUTING.md and a PR template now state the branch contract that had only lived in docs/operations/release.md: routine work targets dev, and main is the release line. A workflow comments on PRs opened against main from anything other than dev or a hotfix/* branch, because GitHub pre-fills the base with the default branch and walks contributors into it.

1.0.1 — 2026-05-27

Release-pipeline validation. No product change — the published binary, REST surface, MCP wire shapes, and audit semantics are identical to v1.0.0. Operators pinning 1.0.0 have no reason to bump; those tracking :latest move forward.

Changed

• Release workflow — arm64 image builds now run on a native arm64 runner (ubuntu-24.04-arm) instead of QEMU emulation. v1.0.0's server image build took 1h24m; this should drop to ~10 min. Multi-arch manifest assembled by a new merge job via docker buildx imagetools create.
• Node 24 opt-in — workflow sets FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true so all actions/* + docker/* run on Node 24 ahead of GitHub's 2026-06-02 forced cutover.

1.0.0 — 2026-05-27

Stability commitment. No code delta since 0.5.0 — this tag marks the point at which the API surface becomes a SemVer commitment.

Changed

• Versioning policy — from this tag onwards every breaking change (REST routes, MCP wire shapes, audit-row JSON keys, database schema, public Rust APIs in published crates) requires a major bump. Operators chasing the :latest tag on the GHCR images can do so without surprise.

Notes

• Docker images cut at this tag receive :latest for the first time — the release workflow suppresses :latest on 0.x and pre-release tags. Pin the version in production rather than tracking :latest unless you have a controlled rollback path.

0.5.0 — 2026-05-26

First public beta. The product surface is stable enough to deploy against, but the API contract is not yet committed — expect breaking changes in 0.6.x and beyond as the run-up to 1.0.0 narrows the surface. Operators running this version against real traffic should pin the image digest and read every subsequent release note.

Highlights

• AI gateway (:3000) — OpenAI / Anthropic / Google / Azure OpenAI / AWS Bedrock with weighted multi-route failover, circuit breakers, semantic response cache (Redis-backed), SSE streaming with PII restore on the wire.
• MCP gateway — per-user OAuth + static-token + admin-shared credential modes, response cache with prefix-based invalidation, per-server circuit breakers keyed by UUID (so a rename or recreate doesn't inherit stale state).
• Audit pipeline — bodies up to audit.body_max_bytes land inline in ClickHouse; oversize bodies offload to S3-compatible object storage (RustFS / MinIO / AWS S3). Request and response bodies are PII-redacted before write; the bucket lifecycle rule is administered from the admin UI (audit.body_s3_lifecycle_days).
• RBAC + identity — JWT access tokens, refresh tokens, OIDC SSO, TOTP, recovery codes. Permissions evaluated on every request from a Redis-cached + DB-backed policy. Per-API-key limits + budgets enforced on the gateway hot path.
• Observability — /metrics (Prometheus exposition, bearer- protected), /api/health (PG + Redis + ClickHouse + S3 deep probe), per-request x-trace-id, structured tracing.
• Admin console (:3001) — React 19 + TypeScript + i18n (en/zh, perfect parity). Dashboard, traces, cost analytics, RBAC editor, MCP server CRUD, settings PATCH.

Quality

• 675 unit + integration tests, gated on make precommit.
• Five rounds of systematic bug audits (≈ 45 bugs fixed, ≈ 800 lines of legacy compat scrubbed) preceded this tag — see commits b0b5820 → dbfe9da for the full series.

Operations

• Helm chart ships an opt-in ServiceMonitor (metrics.serviceMonitor.enabled=true) gated on the auto-generated METRICS_BEARER_TOKEN secret. Pair with kube-prometheus-stack for /metrics scraping.
• deploy/grafana/dashboards/ — starter overview dashboard JSON plus a metric reference + minimal alert rule set in the README.
• docs/operations/secret-rotation.md — JWT_SECRET online rotation and ENCRYPTION_KEY offline re-encrypt procedures.
• docs/operations/backup-restore.md — PG + ClickHouse + S3 procedures with restore-order gotchas, cross-version compat notes, and a quarterly DR drill template.

Known limitations for 0.5.x

• API surface NOT frozen — REST routes, MCP wire shapes, audit-row JSON keys, and database schema may change in any 0.x bump. SemVer kicks in at 1.0.0.
• No online ENCRYPTION_KEY rotation — the documented procedure requires a brief downtime window. Online dual-key rotation is queued for 1.x.

Upgrade path from 0.1.0

The 0.1.0 series was never published; deployments running unreleased builds should: stop the gateway, run db/schema.sql against PostgreSQL, restart against this tag. The schema is idempotent end-to-end, so the apply is safe to repeat.