Work in progress
Speaker-attributed transcription in Rust: scriptrs (Parakeet TDT) for what was said, speakrs for who said it. Both run natively on an NVIDIA GPU, without Python, PyTorch or ONNX Runtime.
DialogPipeline transcribes and diarizes the same audio at the same time, splits
the transcript into words with timestamps, and gives each word the speaker whose
turn it falls in:
use dialogrs::DialogPipeline;
use scriptrs::{LongFormTranscriptionPipeline, ModelBundle, ParakeetDeployment};
use speakrs::{ExecutionMode, OwnedDiarizationPipeline};
let bundle = ModelBundle::from_native_dir("models", ParakeetDeployment::v3());
let transcriber = LongFormTranscriptionPipeline::from_bundle(bundle)?;
let diarizer = OwnedDiarizationPipeline::from_pretrained(ExecutionMode::Cuda)?;
let mut pipeline = DialogPipeline::new(transcriber, diarizer);
// mono f32 at 16 kHz
let dialog = pipeline.run(&audio)?;
for turn in &dialog.turns {
println!("{:?} [{:.1}-{:.1}] {}", turn.speaker, turn.start, turn.end, turn.text());
}dialogrs runs on Linux or Windows x86_64 with an NVIDIA GPU, and needs its
cuda feature:
[dependencies]
dialogrs = { version = "0.1", features = ["cuda"] }To work on dialogrs itself, check out scriptrs and speakrs beside it
(../scriptrs, ../speakrs), which it builds against; the published crate
uses their crates.io releases instead. just check runs the same formatting,
dependency, lint, docs and test checks as CI.
On Windows, the NVIDIA driver provides nvcuda.dll. You also need cuBLAS 12 or
13 and, for speakrs, cuDNN 9. cudakit finds them on PATH or under
%CUDA_PATH%, as its
Windows notes describe. CI
builds and tests dialogrs on Windows, but it hasn't yet run on a Windows GPU.
examples/bench.rs runs a directory of 16 kHz mono wavs, and
bench/whisperx/run.py runs WhisperX on the same files with the same output
layout. bench/sets.py builds the benchmark sets, bench/mixed.sh runs both
pipelines over them, and bench/score.py scores WER, cpWER (speaker-attributed
WER) and DER.
The benchmark mixes six English sets, 212 recordings and 114 hours in all, so no one kind of audio decides the result:
| Set | Recordings | Hours | Audio |
|---|---|---|---|
| Earnings-21 | 44 | 39.3 | earnings calls, up to 20 speakers |
| AMI headset mix | 34 | 18.7 | 3 or 4 person meetings |
| AMI far-field | 34 | 18.7 | the same meetings on one room microphone |
| EdAcc test | 40 | 18.2 | accented English conversations |
| CHiME-6 eval | 4 | 9.7 | dinner parties, far-field, overlapping speech |
| AppTek call center | 56 | 9.4 | customer calls, 4 per accent across 14 accents |
Both pipelines decode English only. On one RTX 5090, end to end (transcription and diarization, audio already in memory), averaged over the six sets:
| Pipeline | WER | cpWER | DER | Speed (x real time) |
|---|---|---|---|---|
dialogrs, Parakeet v3 + speakrs cuda |
20.7% | 28.0% | 21.9% | 1355x |
dialogrs, Parakeet v3 + speakrs cuda-fast |
20.7% | 29.0% | 22.4% | 1617x |
| WhisperX 3.8.6, large-v3 (float16, batch 16) + pyannote community-1 | 23.1% | 30.5% | 21.8% | 77x |
dialogrs transcribes more accurately than WhisperX on every set and attributes words to speakers more accurately (lower cpWER) on every set. Diarization alone (DER) is within a point of WhisperX's pyannote on every set, slightly behind on average.
speakrs cuda is the default. cuda-fast diarizes with a 2 s step instead of
1 s, which makes the whole pipeline about 19% faster but less accurate on every
set except Earnings-21. It also gets the number of speakers right less often:
137 of 212 recordings against 163 for cuda.
| Set | WER dialogrs | WER WhisperX | cpWER cuda |
cpWER cuda-fast |
cpWER WhisperX | DER cuda |
DER cuda-fast |
DER WhisperX |
|---|---|---|---|---|---|---|---|---|
| Earnings-21 | 8.2% | 9.7% | 16.7% | 15.8% | 17.8% | 9.7% | 9.2% | 9.7% |
| AMI headset mix | 21.5% | 23.8% | 27.5% | 28.3% | 30.7% | 17.0% | 17.5% | 17.0% |
| AMI far-field | 23.2% | 28.3% | 30.7% | 31.9% | 35.2% | 19.7% | 20.6% | 19.3% |
| EdAcc | 16.0% | 19.2% | 24.7% | 27.6% | 25.7% | 22.8% | 23.7% | 22.1% |
| CHiME-6 | 44.0% | 44.7% | 54.8% | 55.9% | 58.1% | 49.4% | 49.7% | 49.3% |
| AppTek call center | 11.2% | 12.6% | 13.6% | 14.3% | 15.3% | 13.1% | 13.5% | 13.2% |
Both dialogrs modes share one transcriber, so they have the same WER. WER and cpWER use Whisper's English normalizer. cpWER pools each speaker's words against the reference speakers. DER uses a zero collar and scores overlapped speech.
Wall time for all 212 recordings (114 hours), with WhisperX as 1x:
| Pipeline | Wall time | vs WhisperX |
|---|---|---|
| WhisperX 3.8.6, large-v3 + pyannote community-1 | 1 h 29 min | 1x |
dialogrs, Parakeet v3 + speakrs cuda |
5 min 3 s | 17.7x |
dialogrs, Parakeet v3 + speakrs cuda-fast |
4 min 14 s | 21.1x |