Let plugins provide the transcription backend (custom/self-hosted ASR endpoint) #7807

Describe the problem to be solved

PeerTube's automatic transcription is hard-wired to two local CLI engines, plus PeerTube's own runner:

• video_transcription.engine only accepts whisper-ctranslate2 or openai-whisper. Both are local Python CLIs that the instance (or a runner) installs and runs itself (engine_path, model, model_path).
• With video_transcription.remote_runners.enabled: true, transcription can be moved to a peertube-runner on another machine — but that runner still runs one of these two local CLIs (pip install whisper-ctranslate2, model download, [transcription] section in the runner configuration). It is not a way to use an ASR service that already exists elsewhere.
• There is no plugin hook for transcription. In develop (8.3.0) the server hook list contains ~81 filter and ~28 action hooks and none of them relates to transcription (the closest is action:api.video-caption.created). The server helpers exposed to plugins (serverHelpers.videos.*, database, moderation, …) offer no supported way to attach a caption or transcript either, so a plugin that does its own speech recognition has to write the caption through internal database or filesystem access.

We already run a better ASR engine that PeerTube cannot use. Next to our PeerTube
instance we operate a local NVIDIA Parakeet TDT service on a GPU (RTX 3090) that
exposes an OpenAI-compatible HTTP endpoint (POST /v1/audio/transcriptions). Compared
with the local Whisper engines PeerTube runs, it is measurably better on our language
(German, Hugging Face Open ASR Leaderboard, VoxPopuli test set):

• Lower error rate: Parakeet TDT 0.6B reaches a German WER of 4.12 %, against 6.14 % for Whisper large-v3 — about a third fewer errors, with a model that is 2.5× smaller (0.6B vs 1.5B parameters). PeerTube's default engine (whisper-ctranslate2, model: 'small') is weaker still than large-v3.
• Faster: Parakeet runs at RTFx 3490 on the same leaderboard versus 334 for Whisper large-v3 — roughly 10× faster, i.e. minutes of GPU time instead of hours of CPU time for a long recording.
• Timestamps are part of the model output: it predicts word-level and segment-level timestamps natively, i.e. exactly what the PeerTube transcription widget expects.
• Long-form support and punctuation: up to 24 minutes of audio in a single pass (3 hours with local attention), with automatic punctuation and capitalization; released under CC-BY-4.0.

We cannot use any of that: PeerTube only knows how to install and execute the two
Whisper CLIs. Pointing the instance at our endpoint is impossible, so our instance
keeps running Whisper inside the web container — including a pip3 install of the
transcriber at the start of every transcription job and the model download that
follows — while the GPU that would do the same work ten times faster and with fewer
errors sits idle. That is the concrete gap this request is about: we do not want
another vendor engine in core, we want to be allowed to plug in our own.

The consequence for admins who already operate their own ASR service (whisper.cpp server, faster-whisper-server, Speaches, vLLM, or a GPU machine on the LAN exposing an OpenAI-compatible /v1/audio/transcriptions) is that they cannot use it. Their only options are to let the PeerTube instance (or a container next to it) install Python, whisper and model files and run the inference in-process, or to reimplement the whole flow in a plugin with unsupported database access.

Real-world impact on our instance (Docker, container memory limit 1 GiB): an upload starts transcription at the same time as transcoding, in the same container. Node + ffmpeg + whisper then exceed the cgroup limit and the kernel OOM killer terminates processes inside the container — the main process at 16:52:42 and ffmpeg at 16:56:55 on 2026-10-03 (kernel oom-kill entries, Memory cgroup out of memory: Killed process (peertube) / (ffmpeg)). The container restarts, the transcoding job dies with UnrecoverableError, the video stays in "transcoding failed" and the transcription job then fails with ENOENT …-1080.json. Transcription has produced no caption on that instance since mid-September. We already run a GPU machine with an ASR service — we would simply like PeerTube to call it instead of running whisper in the web container.

You already pointed in this direction when closing #7698 (TwelveLabs Pegasus transcription engine):

Prefer: To extend PeerTube plugin API so admins can install plugins that add transcription capabilities
Or develop a program that uses the PeerTube runners API …

This issue is a request to do exactly the first part, keeping vendors out of core.

Describe the solution you would like

A minimal, opt-in extension of the plugin API. Either of these would already unblock the use case — both is of course fine:

1. Let a plugin provide the transcriber. For example a filter hook such as filter:transcription.get-transcriber.result, or the ability for a plugin to register an engine name that video_transcription.engine can select. The plugin would implement the existing transcription contract (input: path of the audio/video file to transcribe; output: a transcript file plus detected language), so a plugin can call any HTTP endpoint it likes. Server plugins run inside the PeerTube process, so returning an object implementing the existing AbstractTranscriber should not raise serialization problems. The built-in engines must remain the default when no plugin handles the hook.

2. A supported helper to attach a caption file, e.g. serverHelpers.videoCaptions.add({ videoId, language, vttContent }). Combined with the existing action:api.video.uploaded hook, a plugin could run its own ASR and then attach the resulting caption through a documented API instead of writing database rows.

With either option core stays vendor-neutral and admins with their own infrastructure (GPU machine, existing HTTP ASR endpoint) can plug it in.

Beyond our own Parakeet container, a second example of an endpoint that would fit the same slot: CrispASR (github.com/CrispStrobe/CrispASR, MIT) — one C++/ggml binary that ships 119 speech backends (62 of them TTS engines), including the Whisper family, Parakeet TDT (also the multilingual v3), Canary 1B v2, Qwen3-ASR and Voxtral, plus a persistent HTTP server mode:

crispasr --server -m model.gguf --port 8080
curl -F "file=@audio.wav" http://localhost:8080/v1/audio/transcriptions

So: an OpenAI-compatible /v1/audio/transcriptions endpoint (with /v1/audio/speech for TTS, --server-workers for concurrency, and --list-backends to list what the binary can run) and no Python runtime on the PeerTube host. The admin picks the model that fits the language — a language-specialised model, a larger multilingual one, or a quantised small one on weak hardware — and PeerTube only has to speak HTTP to it. Both our Parakeet service and CrispASR are models of the same idea: the endpoint already exists, the model choice belongs to the admin, and core stays vendor-neutral.

For us specifically, the endpoint would be an OpenAI-compatible POST /v1/audio/transcriptions
served by our Parakeet container; a whisper.cpp server, faster-whisper-server or Speaches
deployment would work exactly the same way. A configuration or plugin setting for the
endpoint URL, the model name and an optional API key would be all we need — and the
instance would stop installing Python packages and downloading models at job time.

Additional context

• We can provide a PR and are happy to follow your preferred design (hook name, option 1 or 2, or a different approach).
• Related issues: Add TwelveLabs Pegasus transcription engine #7698 (external engine rejected, with the suggestion above), Create a hook when a video transcoding job is complete #5343 (hook when a transcoding job is complete), Ability to set ComputeType (int8, float16 etc) for transcription in peertube-runner #7311 and Thread configuration for transcription in runner #7231 (transcription settings for the runner), Allow video owner to start transcription #7031 (allow video owner to start transcription).

Provenance: this issue was written by an AI assistant (Hermes Agent, Nous Research) on behalf of the admin of a self-hosted PeerTube instance, and approved by that admin before posting. The benchmark numbers come from public sources (Hugging Face Open ASR Leaderboard, NVIDIA model cards); the failure description comes from our own instance logs.