bug(voice): agent completion readback unreliable (all agents; worst on stream-json flavors)

[heavygee]

Summary

When a coding agent finishes (ready event), the ElevenLabs voice assistant often does not read back the agent's answer reliably. The user may need to ask multiple times; ConvAI sometimes hallucinates a partial summary.

This affects all agent flavors, but not equally - see below.

Two related gaps in web/src/realtime/hooks/contextFormatters.ts:

1. formatReadyEvent (older behavior): injected text told ConvAI "the previous message(s) are the summary" without embedding assistant text inline. All flavors - ConvAI is pointed at context that may not contain the answer, with no inline fallback.
2. formatMessage: does not format stream-json / codex-family payloads ({ type: "codex", data: { type: "message", message: "..." } }), so onMessages and session history context updates are empty for those sessions. Claude-style content arrays (text blocks, plain strings) do work.

Affected agents (how)

Flavor	Ready inject gap	Live context (onMessages / history)	Typical outcome
Claude	Yes	Usually populated (text blocks reach formatMessage)	Unreliable readback - context may help, but ready inject does not embed text; still fails in dogfood
Codex	Yes	Empty (stream-json not formatted)	Readback usually fails; hallucination risk if user presses for a summary
Cursor	Yes	Empty (same stream-json shape as Codex)	Same as Codex
Gemini, Kimi, OpenCode	Yes	Empty if using codex-family stream-json envelopes	Same as Codex/Cursor
All	Yes	Varies	Even when live context exists, old ready text did not paste the final answer inline

Bottom line: the ready-event bug is universal. Codex/Cursor (and other stream-json flavors) also had silent context - no history and no live updates - so they were hit hardest.

Expected

On ready, voice should receive the last assistant speakable text inline (e.g. wrapped in <text>…</text>) so ConvAI can summarize immediately without the user re-prompting. Stream-json messages should flow through formatMessage for live updates and session history.

Repro

Any flavor (ready gap):

1. Open a session with voice enabled
2. Ask a factual question
3. Agent replies in chat with a substantive answer
4. Voice signals "finished" but does not convey the answer unless the user re-prompts

Codex / Cursor / stream-json flavors (additional context gap):

Same steps; voice also lacks interim agent text in context between messages, so readback fails more consistently and confabulation is more likely.

Evidence

Dogfood on ElevenLabs ConvAI: ready inject appeared as a fake user turn with no embedded assistant text; when context was thin (especially Codex/Cursor sessions), assistant confabulated instead of reading hub message content. A Cursor session with the fix applied showed embedded <text>…</text> at ready and accurate readback one second later (conv conv_4501ksdt0athfhfr189tq3jehkcq).

Suggested fix (minimal)

• extractLastAssistantSpeakable() + embed in formatReadyEvent() (ready hook) - all flavors
• Teach formatMessage() the codex stream-json path so live onMessages context is not empty - Codex/Cursor/stream-json flavors

Might be related to open PR #640 (Codex voice context) - not sure if that covers ready readback specifically.

Draft fix: PR #682

[github-actions]

Thanks, verified against the current code. This matches the existing voice-context path.

ready currently does not include the last assistant text inline: the hook calls formatReadyEvent(sessionId) with only the session id, and the formatter only says the previous messages are the summary. Separately, formatMessage() handles strings, { type: 'text', text }, and Claude-style content arrays after unwrapping role/output envelopes, but I did not find handling for { type: 'codex', data: { type: 'message', message: '...' } }, so that shape can produce no voice context.

The minimal fix area looks like the one you identified: add Codex/Cursor message extraction in the realtime formatter, and make the ready path include the last speakable assistant text instead of relying only on prior context.

Relevant code:

• web/src/realtime/hooks/voiceHooks.ts:146 - onReady() sends only formatReadyEvent(sessionId).
• web/src/realtime/hooks/contextFormatters.ts:94 - formatMessage() starts voice formatting for a single decrypted message.
• web/src/realtime/hooks/contextFormatters.ts:100 - non-array content only supports plain strings or { type: 'text', text }, otherwise returns null.
• web/src/realtime/hooks/contextFormatters.ts:202 - formatReadyEvent() references previous messages but does not embed assistant text.
• cli/src/codex/utils/codexEventConverter.ts:133 - Codex agent_message is converted to { type: 'message', message, id }, matching the shape that the realtime formatter needs to understand once wrapped as Codex content.

---

HAPI Bot

[heavygee]

Thanks for confirming against current code — that matches what I saw in dogfood (especially the codex envelope returning null from formatMessage, and ready not embedding text).

I opened a minimal draft along the lines you outlined: #682

• extractLastAssistantSpeakable() + embed in formatReadyEvent() on the ready path
• same codex message extraction for live onMessages / history context
• Vitest coverage in contextFormatters.test.ts

Happy to rebase or fold into #640 if that is the better home — I might have missed overlap there. Marking the PR ready for review now; appreciate any steer on scope.