Skip to content

Add on-device voice typing to Workshop - #1216

Open
bjesus wants to merge 4 commits into
mainfrom
workshop/voice-typing
Open

bjesus wants to merge 4 commits into
mainfrom
workshop/voice-typing

Conversation

@bjesus

@bjesus bjesus commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Add a mic next to the plus button in the Home and chat composers. While listening, the red mic has an expanding ring that respects reduced motion.
  • Show interim speech inline in muted text. Only confirmed words are saved to the draft or included when sending.
  • Require on-device recognition through the Web Speech API. Hide the mic when the browser cannot require local processing; never fall back to a remote recognizer.

Verification

  • 126 targeted composer and Home prompt tests passed, including no-cloud-fallback and late-transcript coverage.
  • Workshop frontend TypeScript and targeted lint passed (three existing test-file lint warnings).
  • Live microphone verification on Firefox/Linux.

@github-actions github-actions Bot added the workshop/frontend Changes to the Workshop frontend label Nov 18, 2026

@clanky-the-clanker clanky-the-clanker Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by the review of 935ac6b1.

This review, of 8fc10d11

💬 Approved with comments — No correctness, security or performance defects found; one INFO note about footer layout on narrow viewports.

Findings: 1 info (1 is an inline comment on the diff)

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.


📋 Code Quality: 💬 1 info

The dictation flow is sound. One layout note on the composer footer.

Recommendation: 💬 Optional Nits

  • [INFO] Footer left cluster changed from shrinkable to shrink-0 while gaining the mic button; narrow viewports squeeze only the model selector. (packages/workshop-frontend/src/features/chat/composer/ChatComposer.tsx:996)
View full review

The dictation flow is sound. When the chat or draft changes, on send and on edit, the recognition callbacks are invalidated. A recognizer without processLocally never falls back to the cloud.

Footer layout (ChatComposer.tsx:996): the left cluster was min-w-0 flex-1 overflow-hidden and is now shrink-0, and the microphone button now sits in it. The add, microphone and options buttons then need roughly 120–130px on mobile, so on narrow viewports only the model selector can shrink.


✅ All other checks passed

🔒 Security: ✅ LGTM

Voice typing requires processLocally, and the code reads the flag back after setting it, so recognition cannot fall back to a cloud recognizer. Transcript text goes through React rendering, not raw HTML. Nothing new is logged or sent to a new destination.


📖 AGENTS.md: ✅ LGTM

No AGENTS.md concerns.


⚡ Performance: ✅ LGTM

No performance concerns.


📝 Documentation: ✅ LGTM

No documentation concerns.


Reviewed commit: 8fc10d11

{/* Footer row: add, voice typing, and options left; model and send right */}
<div className="flex items-center justify-between gap-1.5 px-3 pb-1.5">
<div className="flex min-w-0 flex-1 items-center gap-1.5 overflow-hidden">
<div className="flex shrink-0 items-center gap-1.5">

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ INFO · Code Quality

Footer left cluster changed from shrinkable to shrink-0 while gaining the mic button; narrow viewports squeeze only the model selector.

The left footer cluster was min-w-0 flex-1 overflow-hidden and is now shrink-0, and the new microphone button sits in it. On mobile the add, mic and options buttons then take a fixed ~120–130px, so at 320–360px with a long model name only the model selector can shrink. It would be worth checking the footer at that width, or keeping min-w-0 on this group.

@github-actions

Copy link
Copy Markdown

Preview: pr1216-workshop-voice-typing

https://pr1216-workshop-voice-typing-router.cloudflare-os-previews.workers.dev

Dashboard · deleted when this PR closes


setStatus("starting");
try {
recognition.start();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please make first-use setup work when the language pack is downloadable. On a fresh Chrome 157 profile, available({ langs: ["en-US"], processLocally: true }) returned downloadable, and clicking the mic reached this call and failed with language-not-supported. Every retry fails the same way because Workshop never invokes the language-pack installation API. Check availability for the language the recognizer will use and offer a user-initiated install() when needed, with local processing still required.

editedText: string,
cursor: number,
) => {
const { editStart, oldEnd, newEnd } = findTextEdit(displayedText, editedText, cursor);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can discard part of an explicit paste. With Build confirmed and a dashboard in the preview, select all and paste Build a better dashboard. I ran this helper and applyComposerTextEdit with that post-paste value and caret; they saved Buildbetter . The diff treats matching preview characters as unchanged even though the user replaced them. Keep the provisional text outside the editable textarea so native replacement keeps what the user pasted. That also avoids losing the existing draft’s undo history, which I reproduced in Chrome by assigning and removing a preview through textarea.value. This select-all replacement is an observable contract worth covering under the frontend guide’s test rule: ‘Test observable contracts, product rules, state transitions, accessibility behavior, race handling, and failure paths.’

/>
<textarea
value={inputValue}
value={displayedText}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep the newest speech visible when dictation runs past the height cap. Clicking the mic leaves the textarea unfocused on desktop Chrome. I checked the four-row resizing behavior there: after appending a preview, the caret was at the end but scrollTop stayed at 0. The mirror follows that scroll position and clips the new words, so the inline feedback disappears on long drafts. Follow the live tail after layout while the user is at the end, without pulling them away if they deliberately scroll up.

@ask-bonk

ask-bonk Bot commented Nov 18, 2026

Copy link
Copy Markdown

Review: 3 findings.

Posted all three findings inline. I verified the browser behavior in Chrome 157 and reproduced the draft-edit bug directly against the changed code.

github run

@clanky-the-clanker clanky-the-clanker Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by the review of 582eeedd.

This review, of 935ac6b1

💬 Approved with comments — No correctness, security or performance defects; one INFO note about the footer layout on narrow viewports is still open.

Findings: 1 info

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.


📋 Code Quality: 💬 1 info

The language-pack download flow is sound; the footer layout note is still open.

Recommendation: 💬 Optional Nits

  • [INFO] Footer left cluster is now shrink-0 and also holds the mic button, so on narrow viewports only the model selector can shrink. (packages/workshop-frontend/src/features/chat/composer/ChatComposer.tsx:996)
View full review

Language-pack download (useComposerDictation.ts): every await after available() and install() re-checks signal.aborted, and start() never runs after a chat switch, a send or a cleanup. Recognizers without available go straight to start(). The tests cover the chat-switch race.

Footer layout (ChatComposer.tsx:996): the left cluster went from min-w-0 flex-1 overflow-hidden to shrink-0 and now also holds the mic button, so on narrow viewports only the model selector can shrink.


✅ All other checks passed

🔒 Security: ✅ LGTM

Recognition still requires processLocally, and the code reads the flag back after setting it. The language-pack available() and install() calls also pass processLocally: true, so there is no cloud fallback. Transcripts are rendered as React text. Nothing new is logged or sent to a new destination.


📖 AGENTS.md: ✅ LGTM

No AGENTS.md concerns.


⚡ Performance: ✅ LGTM

No performance concerns.


📝 Documentation: ✅ LGTM

No documentation concerns.


Reviewed commit: 935ac6b1


setStatus("starting");
try {
session.startRequested = true;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please handle cancellation until the browser confirms startup. With the explicit lang, Chrome 157 checks availability asynchronously inside start() and ignores abort() and stop() until the session starts. After installing the pack in Chrome 157, I verified that both start(); abort() and start(); stop() still emitted start. If the user switches chats in that window, release() removes every listener and resets the mic button, but the old session can still start the microphone. Pressing Stop in the same window leaves the button stuck on ‘Finishing’ while recognition starts. Keep a one-shot startup handler that aborts stale sessions or honors a pending stop, and test with a deferred native start event; the current mock emits it synchronously. The frontend guide says: ‘Effects that fetch or subscribe must clean up stale work and subscriptions.’

@ask-bonk

ask-bonk Bot commented Nov 18, 2026

Copy link
Copy Markdown

Review: 1 findings.

Posted one new inline finding. I verified that the language pack installs on a fresh Chrome 157 profile. The two earlier findings about paste handling and transcript scrolling still apply.

github run

@clanky-the-clanker clanky-the-clanker Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by the review of cff6e5c3.

This review, of 582eeedd

💬 Approved with comments — No correctness, security or performance defects found. There are two INFO notes: the footer can't shrink on narrow viewports, and typing cancels a language-pack download that is still running.

Findings: 2 info (1 is an inline comment on the diff)

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.


📋 Code Quality: 💬 2 info

Dictation logic, abort handling and late-transcript handling look correct; two INFO notes.

Recommendation: 💬 Optional Nits

  • [INFO] Footer left cluster is shrink-0 and holds the mic button, so on narrow viewports only the model selector can shrink. (packages/workshop-frontend/src/features/chat/composer/ChatComposer.tsx:1004)
  • [INFO] Typing while a language pack is downloading silently cancels voice typing setup. (packages/workshop-frontend/src/features/chat/composer/ChatComposer.tsx:620)
View full review

Footer layout (ChatComposer.tsx:1004): the left cluster changed from min-w-0 flex-1 overflow-hidden to shrink-0, and it now also holds the mic button. On narrow viewports, only the model selector can shrink.

Typing cancels a pending download (ChatComposer.tsx:620): handleInputChange cancels dictation in every non-idle state, including downloading and starting. The first keystroke during a language-pack download ends the session without any message. The user then has to click the mic again.

The abort re-checks after each await, the cleanup of the download timeout, the late-result dropping after a send or chat switch, and the mapping of edits from the displayed text back onto the draft all look correct. The tests cover these paths.


✅ All other checks passed

🔒 Security: ✅ LGTM

Recognition still requires processLocally, and the code reads the flag back after setting it. The language-pack available() and install() calls also pass processLocally: true, so there is no cloud fallback. Transcripts are rendered as React text, and nothing new is logged or sent anywhere.


📖 AGENTS.md: ✅ LGTM

No AGENTS.md concerns.


⚡ Performance: ✅ LGTM

No performance concerns. Timeouts and listeners are cleaned up, and the language-pack check runs once each time dictation starts.


📝 Documentation: ✅ LGTM

No documentation concerns.


Reviewed commit: 582eeedd


const handleInputChange = (newValue: string, editCursorPos?: number) => {
const handleInputChange = (newValue: string, editCursorPos: number) => {
if (dictating) dictation.cancel();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ INFO · Code Quality

Typing while a language pack is downloading silently cancels voice typing setup.

dictating is true for every non-idle status, including downloading and starting. The download banner says it may take a couple of minutes, but the first keystroke cancels the session without any message, and the mic never starts. Consider cancelling only while listening or stopping, or telling the user that setup was cancelled.

clearDownloadTimeout();
signal.removeEventListener("abort", clearDownloadTimeout);
}
if (signal.aborted) return;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please cancel the pending mic start if the user switches tabs during the language-pack download. A user with microphone permission already granted can leave after seeing ‘The mic isn’t listening yet’ and have capture begin later in the background. I confirmed that resolving the install while document.visibilityState is hidden still calls start() in this hook, and Chromium’s FrameSessionTracker intentionally allows background recognition on desktop. Let the browser finish the download, but invalidate the pending start on visibilitychange so the user can click the mic when they return. The frontend guide says to ‘Test observable contracts, product rules, state transitions, accessibility behavior, race handling, and failure paths’; cover the install resolving after the tab is hidden.

title={label}
icon={preparing ? (
<span aria-hidden="true" className="text-kumo-subtle">
<Loader size="sm" className="motion-reduce:animate-none" />

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The setup spinner still animates with reduced motion enabled. Kumo’s Loader renders SVG <animateTransform> and <animate> children; motion-reduce:animate-none only affects CSS animation. I confirmed in Chrome that the installed loader’s rotation changes even when the reduced-motion query matches and the SVG’s computed animation-name is none. Hide the spinner under reduced motion and show a static setup icon in its place, so the mic still has a visible preparing state.

@ask-bonk

ask-bonk Bot commented Nov 18, 2026

Copy link
Copy Markdown

Review: 2 findings.

Posted both findings inline.

github run

@clanky-the-clanker clanky-the-clanker Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💬 Approved with comments — There are no correctness, security or performance defects. One INFO note remains: typing during a language-pack download cancels voice typing setup.

Findings: 1 info

The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.


📋 Code Quality: 💬 1 info

No new defects. Typing still cancels a language-pack download that is in progress.

Recommendation: 💬 Optional Nits

  • [INFO] Typing while a language pack is downloading silently cancels voice typing setup. (packages/workshop-frontend/src/features/chat/composer/ChatComposer.tsx:620)
View full review

Typing cancels voice typing setup (ChatComposer.tsx:620): handleInputChange cancels dictation whenever status !== "idle", which includes starting and downloading. A keystroke during a language-pack download ends the session without any message. onCompositionStart (line 844) uses the same condition.

These all look correct:

  • the abort re-checks after each await
  • the download timeout cleanup
  • dropping late transcripts after a send or chat switch
  • mapping edits from the displayed text back onto the draft

One earlier note, about the footer layout on narrow viewports, remains open.


✅ All other checks passed

🔒 Security: ✅ LGTM

Recognition still requires processLocally, and the code reads the flag back after setting it. The language-pack available() and install() calls also pass processLocally: true, so there is no cloud fallback. Transcripts are rendered as React text, and nothing new is logged or sent anywhere.


📖 AGENTS.md: ✅ LGTM

No AGENTS.md concerns.


⚡ Performance: ✅ LGTM

No performance concerns.


📝 Documentation: ✅ LGTM

No documentation concerns.


Reviewed commit: cff6e5c3

seedNonce,
onTranscript: (text) => {
recordDraftEdit();
setInputValue((draft) => appendDictatedText(draft, text));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Preserve the textarea selection when appending finalized speech. In Chrome 157 I selected Build in Build a dashboard while listening, then delivered a final result of with filters with no interim preview. The selection jumped from [0, 5] to [30, 30], so typing Create produced Build a dashboard with filtersCreate instead of replacing Build. This still happens if the interim preview is moved out of the textarea. Preserve the selection in the confirmed draft, advancing the caret only if it was already at the end. A browser test for this replacement would cover the observable behavior and race handling called out in the frontend guide.

@ask-bonk

ask-bonk Bot commented Nov 18, 2026

Copy link
Copy Markdown

Review: 1 findings.

Posted one new finding inline, reproduced in Chrome 157.

github run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

workshop/frontend Changes to the Workshop frontend

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant