feat(asr): 接入阶跃星辰 StepAudio 2.5 批式转写,词典走一等 hotwords - #835
Merged
Conversation
StepFun 的 /v1/audio/transcriptions 是标准 OpenAI multipart(实测 2026-07), 归入 Whisper 兼容路由即可,无需新协议客户端: - 前端新增 stepfun preset(https://api.stepfun.com/v1 + stepaudio-2.5-asr), i18n 五语言补齐; - StepFun 静默忽略 prompt(实测返回 200 但不参与偏置),词汇偏置走专门的 hotwords 参数(JSON 数组字符串)——WhisperBatchASR 新增 with_hotwords, wiring 按 whisper_uses_hotwords 在 prompt / hotwords 间二选一路由, 用户词典在 StepFun 下才真正生效; - response_format 只认 json/text → verbose_json 幻听过滤保持关闭; - 修「验证」按钮对静音敏感厂商的假阴性:探针是 250ms 纯静音,StepFun 返回 400 "no speech found",但走到内容拒收说明鉴权(错 key=401)与模型名 (错模型=404)均已通过——判为验证成功而非凭据错误。 实测矩阵(真实 key 逐项打点):真实语音+json/缺省/带 prompt/带 hotwords 均 200 且转写准确(5.5s 音频约 0.95s 往返);verbose_json 400;静音 400 no speech;错 key 401;错模型 404。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
PR Reviewer Guide 🔍(Review updated until commit 1a91d88)Here are some key observations to aid the review process:
|
照百炼「一个入口、按模型二次路由」的先例:stepfun preset 下模型填
`stepaudio-2.5-asr-stream`(或任意 `*-stream`)即走新的实时 WS 客户端
(asr/stepfun_realtime.rs),其余模型维持批式 Whisper 兼容路径。凭据与
「已配置」判定仍看真实 active `stepfun`,一把 key、同一个 https base
(wss URL 由 client 自动派生)。
协议与 qwen_realtime 同为 OpenAI Realtime 风格,但四处关键差异
(2026-07-16 真实接口逐项实测确认):
- 模型在 session.update 里传(session.audio.input.transcription.model),
session 配置是 audio.input.{format,transcription,turn_detection} 嵌套形状;
- delta 的 text/stash 是「已确定前缀 + 未定尾巴」的拼接关系(Qwen 是互斥取一);
- 没有服务端结束事件:session.finish 回 transcript.response.error,server_vad
下 commit 被静默忽略。收尾靠客户端状态机:松手时冲刷尾音频 + 补送 700ms
静音帧逼 VAD 关段,finishing && open_segments==0 即完成;纯静音会话由
1.2s 宽限期兜底空文本成功(验证按钮复用该路径,全程 ~2s);
- 实时协议词汇偏置走 transcription.prompt(批式恰相反只认 hotwords),
词典按 effective 协议自动选通道。
真机端到端(真实 key、模拟 recorder 时序):session 建立 186ms,
松手→最终结果 251ms(批式 ~950ms),5.5s 中文语音转写一字不差,
多句段 VAD 断句由 join_segments 复用 qwen 拼接规则。
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
Persistent review updated to latest commit 4fae410 |
Contributor
|
Persistent review updated to latest commit fa06797 |
Contributor
|
Persistent review updated to latest commit 1a91d88 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
User description
概要
接入阶跃星辰(StepFun)StepAudio 2.5 ASR 作为一等供应商。它的
/v1/audio/transcriptions是标准 OpenAI multipart 形状,归入现有 Whisper 兼容路由即可——无新协议客户端,改动集中在 preset 注册和两个小适配。改动
ASR_PRESETS新增stepfun(https://api.stepfun.com/v1+stepaudio-2.5-asr),i18n 五语言补齐。hotwords:实测 StepFun 静默忽略prompt(200 但不参与偏置),词汇偏置要走它专门的hotwords参数(可解析的 JSON 数组字符串)。WhisperBatchASR新增with_hotwordsbuilder(沿用with_request_format的模式),wiring 按whisper_uses_hotwords在 prompt / hotwords 间二选一路由,QA 与听写两处构造点共用whisper_vocab_for_provider。response_format只认json/text,发verbose_json会 400(照 siliconflow / zhipu 先例)。"no speech found"——但能走到内容拒收说明鉴权(错 key 是 401)和模型名(错模型是 404model_invalid)都已通过、转写管线是通的。现在把这类 400 判为验证成功,不再对静音敏感的厂商恒报假阴性。匹配措辞刻意收窄(仅no speech),宁可漏判也不把真正的参数错误误判成功。实测矩阵(真实 StepFun key 逐项打点,2026-07-16)
response_format=json/ 缺省 / 带prompt/ 带hotwordsresponse_format=verbose_jsononly support json,textno speech foundGET /v1/models(拉取模型按钮)测试
asr::whisper::tests::hotwords_sent_as_json_array_field_without_prompt(本地 TCP 服务器断言 multipart 含hotwordsJSON 数组、不含prompt、空白词条被过滤)coordinator::tests::stepfun_is_whisper_compatible_with_hotwords_vocab(路由/格式/切分/词典二选一契约)commands::providers::tests::silence_probe_content_rejection_is_not_a_credential_errortsc --noEmit/cargo check通过。后续(不在本 PR)
stepaudio-2.5-asr-stream走 OpenAI Realtime 风格 WS(input_audio_buffer.append事件族,与 qwen_realtime.rs 同源),如需边说边出字可照百炼先例在同一 stepfun 入口下按模型名二次路由;批式 0.15 元/时 vs 流式 1.2 元/时,听写场景批式已足够(松手 ~1s 上屏)。🤖 Generated with Claude Code
PR Type
Enhancement
Description
Add StepFun StepAudio 2.5 ASR support
Realtime WebSocket client (stepfun_realtime.rs)
Batch ASR via Whisper-compatible endpoint with hotwords
One-entry dual-protocol routing (model suffix -stream)
Frontend preset and i18n, validation fix for silence probe
Diagram Walkthrough
File Walkthrough
9 files
Register stepfun_realtime moduleAdd StepFun realtime WS ASR clientAdd hotwords builder and encodingRoute StepFun validation to realtime or batchAdd StepfunRealtime variant and routing logicWire StepFun with hotwords and dual protocolIntegrate StepFun realtime into dictation flowHandle StepFun realtime in QA session endCancel StepFun realtime in resource cleanup1 files
Make join_segments pub(crate) for reuse5 files
Add English i18n for StepFun presetAdd Japanese i18n for StepFun presetAdd Korean i18n for StepFun presetAdd Chinese (Simplified) i18n for StepFunAdd Chinese (Traditional) i18n for StepFun1 files
Add StepFun ASR preset definition