Skip to content
This repository was archived by the owner on Jun 17, 2026. It is now read-only.

docs: add native C++ export engine architecture plan #678

Open
kaili-yang wants to merge 4 commits into
base: main
Choose a base branch
from
Open

docs: add native C++ export engine architecture plan #678

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
959d3e9
perf: export phase 1 — hw encoder, latency mode, texture reuse
kaili-yang Jun 1, 2026
6505187
Revert "perf: export phase 1 — hw encoder, latency mode, texture reuse"
kaili-yang Jun 1, 2026
b7c437b
docs: add CapCut prior art section to native C++ export plan
kaili-yang Jun 1, 2026
fb57d5d
docs: rename export plan to export-optimize-native-cpp-plan
kaili-yang Jun 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
157 changes: 157 additions & 0 deletions docs/export-optimize-native-cpp-plan.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,157 @@
# Native C++ Export Engine — Goals & Architecture

**Status**: Draft
**Date**: 2026-06-01

---

## Prior Art — CapCut Desktop

The architecture in this document draws on the export design of [CapCut](https://www.capcut.com) (desktop), which achieves near-instant export on consumer hardware through three interlocking strategies:

- **Full-stack hardware acceleration.** CapCut routes decode, composite, and encode through the platform's native multimedia APIs (Apple VideoToolbox on macOS, NVENC/AMF/Quick Sync on Windows) so the CPU is only coordinating, not processing pixels. The rendering pipeline uses Metal / Vulkan / OpenGL shaders rather than software compositing.
- **Background pre-render cache.** While the user scrubs the timeline and previews effects, CapCut silently renders affected segments into a low-bitrate segment cache. When export is triggered, cached segments are assembled directly without re-rendering — reducing the export to a mux-and-encode pass over pre-computed frames.
- **On-demand decode.** Only frames that survive trim boundaries are decoded. The demuxer seeks to the nearest keyframe before each active segment and skips the rest at the packet level, so a 10-minute source with a 30-second active region decodes approximately 30 seconds of video, not 10 minutes.

The architecture proposed here applies the first and third strategies directly. The second (segment pre-render cache) is a longer-term addition that can layer on top once the core C++ pipeline is in place.

@davideme davideme Jun 2, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why C++? I'm not against it, I see no argument in favor or against it.


---

## Goal

Reduce MP4/GIF export time from minutes to seconds for typical recordings. The target is to reach near-real-time export speed (export faster than the recording duration) on modern consumer hardware with hardware acceleration available.

Secondary goals: eliminate UI jank during export, remove the real-time audio bottleneck, and give the product full control over codec quality and bitrate tuning.

---

## Why the Current Approach Has a Hard Ceiling

The existing pipeline runs entirely inside Chromium's renderer process using the WebCodecs API. This creates structural limits that cannot be fixed with incremental JS-level changes:

- **No hardware control.** The browser decides whether to use a hardware encoder. There is no way to explicitly target NVENC, VideoToolbox, or Intel Quick Sync, or to pass hardware-specific quality parameters.
- **Single-threaded serial loop.** Every frame is decoded, composited, and encoded one at a time inside a single `await` chain. The compositor blocks the encoder; the encoder blocks the next decode. There is no pipeline parallelism.
- **No GPU zero-copy.** Decoded video frames cannot stay on the GPU between decode and encode. The browser forces a CPU round-trip for compositing, which is especially costly on Linux.
- **Real-time audio constraint.** Audio with speed changes is processed through `MediaRecorder` with real-time playback, meaning a clip with 2× speed still takes the original duration to process its audio.
- **WebCodecs is designed for streaming.** Its API model (optimised for WebRTC latency) imposes constraints — fixed latency mode, limited GOP control, no B-frames — that are the wrong trade-offs for offline batch export.

---

## Architecture: Native C++ Export Helper

The solution follows the same multi-process pattern OpenScreen already uses for `openscreen-screencapturekit-helper` and `openscreen-wgc-capture-helper`. A new standalone C++ binary, `openscreen-export-helper`, takes over the entire encode pipeline.

### Process boundary

```
Electron Renderer ──IPC──► Electron Main ──spawn──► openscreen-export-helper
(React / UI) (Node.js) (C++ encode engine)
```
Comment on lines +46 to +49

@coderabbitai coderabbitai Bot Jun 2, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Nit: add a language tag to fenced diagram block.

markdownlint MD040 is valid here; use something like ```text for the process-boundary diagram. nit, but cleaner CI/docs hygiene.

🧰 Tools
🪛 markdownlint-cli2 (0.22.1)

[warning] 46-46: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/export-optimize-native-cpp-plan.md` around lines 46 - 49, The fenced
diagram block showing the process boundary (the three-line block that starts
with "Electron Renderer  ──IPC──►  Electron Main  ──spawn──► 
openscreen-export-helper") should include a language tag (e.g., use ```text) to
satisfy markdownlint MD040; update that fenced code block to begin with a
language identifier like "text" so the diagram is properly tagged.


The renderer and main process are unchanged from the user's perspective: they display progress, handle cancellation, and write the final file. All pixel work happens in the helper process.

### Why a separate process

- **Crash isolation.** A codec crash or driver fault does not take down the UI.
- **True multi-threading.** The helper can run a decode thread, a composite thread, and an encode thread in parallel — something the JS single-threaded model cannot do.
- **Direct OS API access.** The helper calls VideoToolbox, NVENC, DXGI, and VAAPI directly without browser sandboxing.
- **Consistent with existing codebase patterns.** No new integration model to learn or maintain.

---

## Hardware Acceleration Stack

The helper selects the best available backend at runtime, in priority order:

| Platform | Decode | Composite | Encode |
|---|---|---|---|
| macOS (Apple Silicon) | VideoToolbox | Metal compute | VideoToolbox (H.264 / HEVC) |
| macOS (Intel) | VideoToolbox | Metal compute | VideoToolbox |
| Windows (NVIDIA) | NVDEC | D3D11 compute | NVENC |
| Windows (AMD) | AMF decoder | D3D11 compute | AMF encoder |
| Windows (Intel) | Quick Sync | D3D11 compute | Quick Sync |
| Linux | VAAPI / NVDEC | OpenGL compute | VAAPI / NVENC |
| All (fallback) | FFmpeg software | CPU | libx264 / libx265 |

The critical optimisation at each stage is **GPU zero-copy**: the decoded frame lives on a GPU surface, the compositor reads and writes GPU textures, and the encoder consumes the GPU surface directly — no pixel data crosses the CPU bus until the final muxed file is written to disk.
Comment on lines +64 to +76

@davideme davideme Jun 2, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not using an abstraction like libavcodec library from FFmpeg project? that already maintain all these backend layer.

@kaili-yang kaili-yang Jun 3, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the C++ helper does use FFmpeg (libavcodec / libavformat / libswscale). FFmpeg is the abstraction layer over every hardware backend listed in this document — VideoToolbox, NVENC, AMF, Quick Sync, VAAPI. The question is not whether to use FFmpeg, but where to run it.

There are two realistic ways to run FFmpeg-backed code from an Electron app:

Option A — WebAssembly (ffmpeg.wasm). Compile FFmpeg to WASM and run it inside the renderer or main process. This is a real project and works for simple transcodes. The problem is that WASM runs inside the browser's sandbox, which means it cannot open a VideoToolbox session, cannot acquire an NVENC encoder context, cannot use D3D11VA or VAAPI, and cannot share GPU surfaces with the compositor. Every frame would be a CPU copy. WASM is also single-threaded by default; SharedArrayBuffer threads are available but cannot call native OS APIs. For a pipeline whose entire performance argument is GPU zero-copy and hardware encode, WASM erases the benefit entirely.

Option B — Native process or addon. Compile FFmpeg into a native binary (or N-API addon) that runs outside the browser sandbox with full OS API access. This is what this document proposes, and it is exactly how CapCut, DaVinci Resolve, and every other professional desktop video tool works.

So the stack is: C++ process → libavcodec (FFmpeg) → platform HW API (VideoToolbox / NVENC / VAAPI). FFmpeg is not an alternative to this plan; it is a core dependency of it. The C++ layer exists specifically to host FFmpeg outside the sandbox where it can actually reach the hardware.


---

## What the Helper Does

### Decode
The helper demuxes the source file and sends encoded packets directly to a hardware decoder context. Only frames that fall within the active trim regions are decoded; frames in trimmed gaps are skipped at the GOP level after seeking to the nearest keyframe. This mirrors CapCut's on-demand decode and is the single largest gain for heavily edited projects.

### Composite
Each output frame is assembled as a sequence of GPU shader passes on a render texture:

1. Wallpaper / background fill
2. Video frame (crop + resize)
3. Webcam picture-in-picture (if present)
4. Shadow (pre-baked once per export, not per frame)
5. Zoom / pan transform
6. Cursor overlay
7. Annotations

The shadow pass deserves special mention: the current JS pipeline recomputes a full CSS `drop-shadow` filter on every frame. In the C++ compositor, the shadow mask is a static texture baked once before the first frame and reused for the entire export — making it effectively free.

### Encode
The composited GPU texture is handed directly to the hardware encoder surface. The encoder runs concurrently with the compositor: while frame N is being encoded, frame N+1 is being composited. This pipeline overlap is not possible in the current serial JS loop.

Quality mapping from the existing UI quality selector:

| UI preset | Encoder target |
|---|---|
| Medium | Hardware VBR, CQ ~28, fast preset |
| Good | Hardware VBR, CQ ~23, medium preset |
| Source | Hardware VBR, CQ ~18, slow preset |

### Audio
Audio is decoded, speed-adjusted (using a time-stretch filter rather than real-time playback), and re-encoded fully offline on a separate thread. For a clip with 2× speed applied, audio processing takes half the clip duration — not the full duration as today.

### Mux
Audio and video packets are muxed by DTS into a standard MP4 container with `faststart` (moov atom at the front), ready for immediate playback without post-processing.

---

## Communication Contract

The helper follows the same stdin/stdout JSON contract as the existing native helpers:

- **Input**: a single JSON object passed as a command-line argument describing the full export job (paths, effects, quality, trim, cursor data, etc.)
- **Output**: newline-delimited JSON progress events on stdout (`ready`, `progress`, `done`, `error`)
Comment on lines +121 to +122

@coderabbitai coderabbitai Bot Jun 2, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Input transport is inconsistent and lowkey risky at scale.

Doc says “stdin/stdout JSON contract” but then defines input as a JSON command-line arg. For large jobs (trim maps, cursor/effects payloads), argv length limits (especially on Windows) can fail export startup. recommend making stdin the canonical input channel and keeping argv to a tiny bootstrap token only.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/export-optimize-native-cpp-plan.md` around lines 121 - 122, Update the
docs to make stdin the canonical input channel instead of a large JSON
command-line argument: replace the line describing Input as “a single JSON
object passed as a command-line argument” with language that the exporter reads
the full JSON job description from stdin (and that argv should only contain a
tiny bootstrap token/flags), and keep Output described as newline-delimited JSON
progress events on stdout (`ready`, `progress`, `done`, `error`); explicitly
call out that large payloads (trim maps, cursor/effects) must be provided via
stdin to avoid argv length limits.

- **Cancellation**: SIGTERM

The JS side (`NativeExporter`) wraps this in a thin TypeScript class that translates the existing `VideoExporterConfig` into the helper's JSON format and maps progress events back to the `onProgress` callback. The `VideoExporter` (WebCodecs) remains as a fallback for systems where the helper binary is unavailable.

Comment on lines +123 to +126

@coderabbitai coderabbitai Bot Jun 2, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Cancellation contract needs explicit cleanup + atomic output semantics.

SIGTERM-only is kinda under-specified for long-running encode/mux. Please define required behavior on cancel/error: stop pipeline, delete temp artifacts, and only publish output via atomic rename on success. otherwise users can end up with corrupt/partial files that look “done-ish”.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/export-optimize-native-cpp-plan.md` around lines 123 - 126, Define and
implement an explicit cancellation/error contract for the native helper invoked
by NativeExporter: ensure that on SIGTERM or any encoding/muxing error the
helper stops the pipeline, cleans up temporary artifacts (partial files, temp
dirs), and exits non‑zero; only on successful completion write output to a temp
path and publish the final file via an atomic rename/move into the target path.
Update NativeExporter (the TypeScript wrapper around VideoExporterConfig →
helper JSON) to pass a temp-output path, listen for helper exit codes/events,
delete temp artifacts on non‑zero/error/SIGTERM, and only surface success
callbacks (and call onProgress finalization) after the atomic rename; keep
VideoExporter (WebCodecs) as the documented fallback for systems where the
helper binary is unavailable.

---

## Build & Distribution

Follows the existing native helper conventions:

- Built with CMake, one binary per platform/arch
- Distributed alongside the existing helpers under `electron/native/bin/`
- Statically linked against FFmpeg (stripped to required codecs only) to avoid runtime dependency conflicts
- Must be code-signed and notarised on macOS (same as existing helpers)

---

## Phased Delivery

| Milestone | Scope | Value delivered |
|---|---|---|
| 1 | Helper skeleton: decode → passthrough → encode, no effects | Validates IPC contract; unblocks source-quality exports with no effects |
| 2 | Static effects: shadow, wallpaper, padding, border radius | Covers the most common export configuration |
| 3 | Per-frame effects: zoom, crop, motion blur | Full visual parity for standard editing features |
| 4 | Cursor overlay | Native cursor compositing without JS frame rendering |
| 5 | Webcam PiP + audio | Full feature parity |
| 6 | GIF output | Replaces gif.js; uses FFmpeg palette quantisation |

At each milestone the helper is the primary path for the effect combinations it supports; all others fall back to the existing WebCodecs pipeline.

---

## Expected Outcome

On a machine with hardware acceleration available, the export of a typical 2-minute 1080p recording should complete in under 15 seconds. On machines without hardware support, the software fallback via libx264 (multi-threaded) is still substantially faster than the current single-threaded WebCodecs software path.
Loading