[tilllt]

Description

Add a filter:transcription.get-transcriber.result hook, so that a plugin can provide its own transcriber for video subtitles (automatic and on demand transcription) instead of always using the whisper-ctranslate2/openai-whisper engine bundled with PeerTube.

When a plugin returns a transcriber from this hook, PeerTube uses it for that transcription: the whisper engine is then neither loaded nor installed. Plugins that don't handle transcription return the undefined result they receive, and the instance behaviour is unchanged.

The plugin-facing contract (RegisterServerTranscriber):

{
  name?: string

  transcribe (options: {
    mediaFilePath: string
    language?: string
    transcriptDirectory: string
    signal?: AbortSignal
  }): Promise<{ language: string, path?: string, content?: string }>
}

The transcriber either writes the WebVTT file in transcriptDirectory and returns its path, or returns the WebVTT content and PeerTube writes the file for it (handy for an HTTP ASR service that answers with a transcript).

Why: core stays vendor-neutral (no third-party engine added) while admins can plug the ASR service they already run: their own HTTP endpoint, a GPU machine, a multi-backend service such as CrispASR (119 backends behind an OpenAI-compatible /v1/audio/transcriptions endpoint, MIT), or a model that fits a specific language (for example NVIDIA Parakeet TDT for German).

Has this been discussed before?

Yes: #7807. Adding another engine to core was requested in #7698 and rejected for exactly this reason, with the answer that the plugin API (or the runners API) should be extended instead. This PR follows the plugin API route.

Files

• packages/models/src/plugins/server/plugin-transcriber.model.ts — plugin-facing types (RegisterServerTranscriber, RegisterServerTranscriberResult)
• packages/models/src/plugins/server/server-hook.model.ts — declare the hook
• server/core/lib/video-captions.ts — ask the hook for a transcriber before building/installing the built-in engine, and turn the plugin result into a transcript file
• packages/tests/fixtures/peertube-plugin-test-transcription — test plugin returning its own transcriber
• packages/tests/src/plugins/plugin-transcription.ts — test generating a subtitle with the plugin transcriber (no whisper installation or run involved, so it stays fast)

The cli-plugin suite picks the new test file up automatically (findTestFiles ./packages/tests/dist/plugins html-injection.js).

Checks

• npm run tsc -b server/tsconfig.json — OK
• npm run tsc -b packages/tests/tsconfig.json — OK
• oxlint on the changed files — no warning
• npm run mocha -- --timeout 120000 --exit --bail packages/tests/src/plugins/plugin-transcription.ts against a real instance (PostgreSQL + Redis + ffmpeg) — 1 passing: the uploaded video gets an automatically generated en VTT caption whose content comes from the plugin transcriber, and no whisper installation is triggered. The same test was also run with a plugin transcriber that calls a real OpenAI-compatible transcription endpoint (NVIDIA Parakeet TDT 0.6B v3 ONNX, served by our own ASR service): the caption is the actual model transcript, transcribed in ~1 s for a 13 s clip, and the failure path is loud (a wrong model name surfaced as a job error, not a silent empty caption). The committed fixture deliberately stays a stub so the test needs no network access.

I'm happy to adjust the design (hook name, params, contract) or to implement only option 2 of #7807 instead, if you prefer another shape.

Provenance: this PR was written by an AI assistant (Hermes Agent, Nous Research) on behalf of an instance admin, and reviewed by them before posting.

[Chocobozzz]

Hi,

You don't prefer to support an external openapi compatible endpoint instead?

[tilllt]

Hi,

You don't prefer to support an external openapi compatible endpoint instead?

The reason why an OpenAI compatible endpoint (maybe additionally) makes sense, ist that there are virtually hundreds of tools that have settled in the OpenAi/whisper compatible format. I personally would prefer something not in any way related to OpenAi, its more "practicality" when suggesting this. Basically crafting an OpenApi standard for transcription would be great and desirable but "too much work" (IMHO). I guess the maintainer of crispASR could be a person to team up with when establishing an OpenApi based definition...

[Chocobozzz]

Thank you for your answer. Is this PR still a draft?

[tilllt]

Thanks for the ping. It was still a draft for two reasons: the design question was open, and the branch had drifted 112 commits behind develop with a conflict in server/core/lib/video-captions.ts — a review at that point would have been against a stale tree. Both are dealt with now: I rebased it onto develop (5a49790) and took it out of draft (head e2c12f5).

What changed in the rebase:

• the only conflict was an import line in server/core/lib/video-captions.ts (develop added storeHLSFileFromContent): both imports kept, the hook usage itself unchanged
• no functional change to the hook: filter:transcription.get-transcriber.result, and the built-in engines remain the default when no plugin handles it

Verified locally on the rebased head:

• build:server and build:tests (tsc) pass
• oxlint --import-plugin --promise-plugin --node-plugin clean, swagger-cli validate valid, generate-config-schema produces no diff, validate-config-schema OK
• CI on the new head is running while I write this (15 checks)

On the question from October 5: I kept the generic hook as the primary mechanism on purpose. An OpenAI-compatible POST /v1/audio/transcriptions endpoint as a core engine would work for us as well (it is exactly what our own ASR service speaks), but it puts a vendor-shaped contract into the core config, while the hook lets the same endpoint be used by a plugin and also covers the second half of the problem, attaching the produced caption through a supported path instead of database rows.

If you would rather have the OpenAI-compatible engine in core as an additional option — not instead of the hook — I can add it to this PR: it is a small transcriber implementing the existing AbstractTranscriber contract, so an admin could point video_transcription at an endpoint without installing a plugin at all. Your call.

Would you like the hook name or the returned object shape changed? Happy to adjust either.

Drafted with assistance from an AI agent (Hermes, Nous Research) on behalf of the contributing user; the checks above were run locally by rebasing this branch onto develop 5a49790 and building the server and the test suite there.

[tilllt]

Hey, I asked Hermes to rebase the PR against the current state of the dev branch so "in theory" it should me reviewable & mergeable now.

[Chocobozzz]

Thank you.