feat(voice): retire the conversational STT->LLM->TTS loop - #2398
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
The audio-native path is merged end to end — server call plane (#2393), server behaviour (#2395), browser + lifecycle (#2396), UI (#2397) — so the old pipeline is dead weight on the conversational path. This removes exactly that much and nothing else. WHAT WENT, and why each piece could not stay: - `useVoiceMode` + `/api/voice/transcribe`: the loop itself. Whisper existed to turn audio into text before inference; the realtime session hears the audio directly, so there is nothing left for it to do. Verified with a repo-wide search that no other caller reaches the route. - `useVoiceModeStore` and everything reading it — `VoiceCallPanel`, `VoiceModeSettings`, `VoiceModeBorder`, and the mic button in the chat box's footer. The store's only writer was that button. Left in place, the border would be UI that can never render and the button an affordance that toggles a mode nothing implements. The way into voice is the nav-bar trigger. - `selectVoiceStreamText`, `selectVoiceActivationBaseline` and `selectPostBaselineAssistantMessage`: three pure selectors whose only job was deciding which written reply the old path should speak. Spoken turns now arrive as ordinary messages, so nothing derives a "what to say out loud" from the message list any more. - The Whisper rate in `voice-pricing`, and `VOICE_HOLD_ESTIMATE_CENTS` — the flat hold that existed because STT could not know its own cost until the provider answered. Both had exactly one caller, the deleted route. WHAT DELIBERATELY STAYED. `/api/voice/synthesize`, the tts-1/tts-1-hd rates, `estimateVoiceHoldCents`, `VOICE_MAX_INFLIGHT` and `chunkForTts` all back Read Aloud, which is an open PR (#2173) and a genuinely different feature: an audio-native conversation does not replace "read this to me". `mic-errors` stays because the realtime path is now its only consumer. `chunkForTts` is kept despite having no in-tree caller on this branch — its last one went with `VoiceCallPanel` — because `useReadAloud` imports `flushForTts` from it on #2173. Deleting it would break work in flight. knip does not report it, so it needed no ignore. The one knip.json line added is for `@radix-ui/react-slider`: deleting VoiceModeSettings left `components/ui/slider.tsx` as its only importer, and `src/components/ui/**` is already ignored. TESTS DELETED WITH THEIR SUBJECTS, never to make the gate pass: `useVoiceModeStore.test.ts`, `transcribe/route.test.ts`, and the three stream selector tests. `voice-pricing.test.ts` loses its Whisper describe block; the unknown-model and 1¢-floor assertions are kept, retargeted off `whisper-1`. `whisper-1` survives as a fixture in the admin billing-coverage tests, where it stands for historical usage rows that still exist in the database. Gate: monorepo `bun run typecheck` 17/17, `bun run lint` 15/15, knip ratchet green with an unchanged baseline. Unit suites pass (lib 9175, web 17057); the only red files are the DB-backed integration tests, which need a Postgres this worktree has no access to. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYjZgYpebAVgBq9f5oVt81
f00d49c to
b0657cc
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f00d49c48f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| - **Voice mode is now audio-native, and the old hands-free mic in the chat box is gone** — talking | ||
| to an assistant used to mean recording a clip, having it transcribed into text, and having the |
There was a problem hiding this comment.
After this change, the primary README still advertises voice mode as “Speech-to-text + text-to-speech chat workflows” (README.md:264), and apps/web/.env.example:57 still says the OpenAI key powers voice through STT/TTS. Since this commit deletes the transcription endpoint and declares audio-native realtime voice to be the only conversational path, those setup and feature descriptions now misrepresent the shipped architecture; update them alongside this changelog entry.
AGENTS.md reference: AGENTS.md:L228-L228
Useful? React with 👍 / 👎.
1 participant
Chunk C-E. The audio-native path is merged end to end — server call plane (#2393), server behaviour (#2395), browser + lifecycle (#2396), UI (#2397) — so the Whisper -> text LLM -> tts-1 loop is dead weight on the conversational path. This deletes exactly that, plus one changelog entry. No refactors, no behaviour changes beyond the removal itself.
The distinction that matters
Two different features shared the
/api/voice/*namespace. Only the conversational one goes.useVoiceMode.ts->/api/voice/transcribe(whisper) -> chat stream ->/api/voice/synthesize-> hand-rolled VAD/barge-in/api/voice/synthesize, open PR #2173)/api/voice/synthesizestays, with its route tests passing (4/4). The tts-1 and tts-1-hd rates,estimateVoiceHoldCents,VOICE_MAX_INFLIGHTandchunkForTtsall stay for the same reason.#2173 has not landed on master (still open against
master,pu/read-aloud), so this deletes against the tree as it is today, rebased onto the currentpu/gpt-realtimetip (0b7159a). That leaves one thing for #2173 to resolve on rebase, called out below.Acceptance criteria
No references to
useVoiceModeor/api/voice/transcriberemain. Asserted by repo-wide search over the whole tree (excludingnode_modules,.next,.git):Both comments that formerly named the retired module by name were reworded to describe the loop rather than the symbol, so the search is clean rather than clean-except-for-prose.
Read Aloud still works end to end.
/api/voice/synthesizeand its tests are present and green;chunkForTts.ts(whichuseReadAloudimportsflushForTtsfrom) is present with all 53 of its tests green.readAloudPlayer.tsitself is not on this branch — it does not exist onpu/gpt-realtimeormaster, only on the unmerged #2173. So "readAloudPlayer present" cannot be asserted here; what is asserted is that nothing this PR touches removes anything it imports.No skipped or weakened tests. Every deleted test went with its subject:
stores/__tests__/useVoiceModeStore.test.tsuseVoiceModeStore.tsapi/voice/transcribe/__tests__/route.test.tstranscribe/route.tsstreams/__tests__/selectVoiceStreamText.test.tsselectVoiceStreamText.tsstreams/__tests__/selectVoiceActivationBaseline.test.tsselectVoiceActivationBaseline.tsstreams/__tests__/selectPostBaselineAssistantMessage.test.tsselectPostBaselineAssistantMessage.tsvoice-pricing.test.tsloses only itscalculateVoiceCostDollars — Whisper STTdescribe block and thewhisper-1 = $0.006/minrate assertion, both of whose subject is the deleted rate. The two remainingwhisper-1mentions in that file were assertions about unknown-model behaviour that merely happened to use it as the example (estimateVoiceHoldCents1¢ floor;calculateRealtimeCostDollarsbilling 0 for a non-realtime model) — those are kept, retargeted onto a still-live model string, so the behaviour stays covered.knip does not regress.
bun run knip:check→[ok] knip: 4 issue(s), all within baseline (4).knip-baseline.jsonis unchanged. Oneknip.jsonline was added, explained below.What went, and why nothing could stay behind
useVoiceMode.ts+/api/voice/transcribe— the loop. Whisper existed to turn audio into text before inference; the realtime session hears the audio, so there is nothing for it to do. Repo-wide search confirmed the route had exactly one caller.useVoiceModeStoreand its whole dependent set —VoiceCallPanel,VoiceModeSettings,VoiceModeBorder, and the hands-free mic button inInputFooter. The store's only writer was that button (viahandleVoiceModeToggleinSidebarChatTab/GlobalAssistantView). Keeping the store would leaveVoiceModeBorderas UI that can never render (isEnabledpermanently false) and the footer button as an affordance that toggles a mode nothing implements. The entry point to voice isVoiceNavTriggerin the top bar.selectVoiceStreamText,selectVoiceActivationBaseline,selectPostBaselineAssistantMessage— three pure selectors whose sole job was choosing which written reply the old path should speak aloud. Spoken turns are now ordinary messages written by the realtime server, so nothing derives "what to say out loud" from the message list any more. Zero other callers.voice-pricing.tsandVOICE_HOLD_ESTIMATE_CENTS— the flat hold that existed precisely because STT could not know its own cost until the provider answered. One caller each: the deleted route.VOICE_MAX_INFLIGHTstays (synthesize passes it).Cross-references in prose that pointed at deleted files were repointed, not left dangling:
tool-runners.tsnow citesapps/realtime/src/voice/call-metering.ts(the live flat-estimate hold -> settle protocol), the realtime call route's header now cites../synthesize/route.ts, andMACHINE_HOLD_ESTIMATE_CENTSnow mirrorsREALTIME_SESSION_HOLD_ESTIMATE_CENTS.Two things worth a reviewer's attention
1.
@radix-ui/react-sliderandchunkForTtsare knip-ignored, not deleted.Deleting
VoiceModeSettings(the TTS speed slider) leftcomponents/ui/slider.tsxas the only importer of@radix-ui/react-slider, andsrc/components/ui/**is already knip-ignored — so knip reported the dependency unused. Added toignoreDependenciesalongside@radix-ui/react-navigation-menu, which is there for exactly this reason. Deleting a shadcn primitive is out of scope for a deletion-only chunk.chunkForTts.tsneeded no ignore in the end — knip does not report it — but it is worth stating that it was deliberately kept despite having no in-tree caller on this branch:useReadAloudimportsflushForTtsfrom it on #2173. Deleting it would break work in flight.2. #2173 will hit one conflict on rebase, by design.
readAloudPlayer.tsonpu/read-aloudimportsuseVoiceModeStoreto make Read Aloud mutually exclusive with a live voice-mode call. That store no longer exists. The mutual exclusion still matters — Read Aloud's own TTS output must not be picked up by a live call's microphone — but it now has to key off the realtime session (VoiceSessionContext) instead of the retired store. That is a one-import change in #2173, and preserving a dead store to defer it would have meant shipping the exact thing this chunk exists to remove.Gate
bun run typecheck(monorepo root) — 17 successful, 17 total. Confirmed against directcd apps/web && bunx tsc --noEmit→ exit 0, after clearing the pre-seeded.next/typesentry for the deleted route and regenerating with a realnext build.bun run lint— 15 successful, 15 total.bun run knip:check— green, baseline unchanged.packages/libunit tests — 9175 passed, 0 failed.apps/webunit tests — 17057 passed, 10 failed.Every failing test file in both suites is DB-backed (
ensureTestDb/requireDb→ECONNREFUSED 127.0.0.1:5432) — this worktree has no test Postgres. None of them touch voice.