Skip to content

Latest commit

 

History

History
422 lines (350 loc) · 23.2 KB

VIDEO-MPV.md

File metadata and controls

422 lines (350 loc) · 23.2 KB

mpv Video Backend & Layered Chroma-Key Compositing

The optional video backend is libmpv, dlopened at runtime, and on top of it a transparent-colour (chroma-key) stage: the user picks a colour on a video and sees through it to whatever is beneath, including other videos that are themselves keyed. Layers are separate transparent viewer windows, stacked and composited by the window server rather than inside the app.

← Feature notes index · Viewer · Architecture · TODO

What is built

Phases 0–4 shipped (2026-06-23), all compile-green; see commits + the NOTES.md decision log. At a glance:

  • 0 spike: SW render emits real alpha ✅
  • 1a ferail-video-mpv crate to parity ✅
  • 1b optional backend swapped to mpv, ferail-video-vlc deleted ✅
  • 2 live set_enhance, VLC-era reopen apparatus removed ✅
  • 3 single-layer chroma key + eyedropper ✅ (screenshot-verified)
  • UI professional popup rework (labeled sections, swatch/hex/Pick) ✅
  • fix vo=libmpv: mpv was opening its own native window + deadlocking ✅
  • 4 transparent stacking windows (replaced the in-app layer stack) ✅
  • 5/4b: deferred items below (rename: done)

Compositing model: transparent stacking windows (2026-06-23). The first Phase 4 cut composited background videos inside one window (a Vec<BgLayer> stack with a "Layers" popup section). That was the wrong model, per user direction, the real design is one keyed video per window, and the windows themselves are transparent, so independent viewer windows stack and the OS window server composites them on the GPU. So the in-app layer stack was removed and replaced by a Transparent toggle in the viewer toolbar (window.set_background_appearance(Transparent) + transparent root/stage backgrounds). With it on, the chroma-keyed-transparent video pixels and the window background are see-through, so stacking two viewer windows (pair with Stay on top) shows the lower video through the upper's keyed regions: composited by macOS, no app-side compositing. This also answers "rendering on the GPU": layer compositing is now fully GPU via the window server. The remaining CPU cost is the per-window SW frame pull; eliminating that (mpv GL render → IOSurface → gpui::surface) is the documented escalation, still open.

The earlier deadlock fix: without an explicit vo, mpv created its native macOS gpu (Cocoa) video output: a separate "- mpv" window with a CVDisplayLink that dispatch_syncs to the main thread, deadlocking on teardown. vo=libmpv makes the SW render context the only output (no window).

Phase 3: single-layer chroma key: a "Transparent colour" section in the adjustments popup (mpv video only) with an on/off toggle, an eyedropper swatch (click it, then click the video to sample the key colour from the live frame), and Similarity (range width) + Blend (edge feather) sliders. The key is pushed live via set_chroma_key, so keyed pixels arrive transparent and the stage background shows through. No new icon (the swatch is the eyedropper affordance), so ICONS.md is untouched. cargo check -p ferail-gpui --features mpv is green.

Verified by screenshot + UI reworked. The adjustments popup renders correctly with the full mpv-video control set (a new screenshot-only --viewer-adjust-video fixture forces the panel without a live stream, since the real frame-pull poll never lets a headless render settle). The popup was then reworked to look more professional: labeled Colour / Enhance / Transparent colour sections (replacing bare separators), and the key picker now shows a swatch + hex readout + a clear Pick button (arms the eyedropper) instead of a tiny "click swatch" hint. Still pending: a screenshot of a live keyed video (the keyed pixels actually transparent), which needs the poll to run: a follow-up.

Per-window mute shipped, each viewer window has a mute toggle in the video transport (volume-x/volume-2 button → set_muted), muted by default, so stacked transparent viewers never all play audio at once; sound is opt-in per window. (An earlier focus-follows-audio version: auto-mute while the window wasn't active: was replaced by the explicit toggle.)

GPU video path - spiked, not viable (keep the SW path)

The idea was to skip the per-frame CPU readback/upload by handing mpv's frame to gpui's surface element (gpui::surface(CVPixelBuffer), GPU-composited via Metal). The spike killed it for our case:

  • gpui's surface fragment shader (gpui_macos/.../shaders.metal, surface_fragment) is hardcoded for opaque bi-planar YUV (NV12): it samples a Y plane + a CbCr plane, does YCbCr→RGB, and returns alpha = 1.0. So a surface cannot carry the chroma-key transparency: the keyed alpha would be discarded, breaking the transparent-window feature (the whole point).
  • gpui is a pinned git dep with no [patch]/fork; this repo forks small pieces locally rather than the engine, so patching the surface shader to accept BGRA-with-alpha isn't on the table.

Conclusion: the SW RenderImage path stays: it carries alpha, so it's the only path that supports keyed/transparent video, which is the headline feature. The per-frame upload is the accepted cost. Note the layer compositing is already fully on the GPU (the OS window server composites the stacked transparent windows), so the only remaining CPU work is each window's frame rasterize+upload. A future opaque-only fast path via gpui::surface (YUV, no key) is possible if non-keyed playback ever becomes a bottleneck, but it'd be a second code path for marginal gain.

Phase 2 recap: removed the VLC-era seamless-reopen apparatus: commit_video_enhance now pushes filters through live set_enhance; the video_pending_seek/video_repause deferral and the poll's pre-seek-frame dance are gone (−67 lines in window.rs). The decision log is in NOTES.md (2026-06-23 entry).

Identifier rename to mpv - done

Every user-facing vlc identifier was renamed to mpv:

  • the cargo feature vlc → mpv (build with --features mpv),
  • the persisted video_backend value "vlc" → "mpv" and field vlc_app_path → mpv_path (the Settings "mpv library" input),
  • the Settings → Plugins dropdown label "VLC" → "mpv", plus all the internal symbols (resolve_mpv_pref, default_mpv_path, MPV_VIDEO_EXTS, …) and comments.

Back-compat: app_state still accepts a legacy video_backend=vlc on load and normalises it to mpv, so existing configs keep selecting the provider; the next settings save writes mpv.

Phase 0 findings (verified, not assumed)

  • SW render → BGRA pull works, same shape as libvlc's vmem, so the copy_frame → (w,h,BGRA) seam is untouched.
  • THE GATE: SW render emits a real alpha channel. PASS. A live colorkey filter produced correct per-pixel alpha through SW render (keyed background transparent, foreground opaque: screenshots/mpv-probe-B_alpha_green.png). Keying lives in mpv's filter chain, live and off our threads, not a CPU pass. Recipe: end the vf chain in an alpha format + request bgra: vf = lavfi=[…,format=rgba,colorkey=color=0xRRGGBB:similarity=…:blend=…].
  • Live vf change applies with no re-open → the VLC reopen apparatus goes.
  • Correction: mpv's brightness/contrast/… equalizer properties are a no-op on the SW-render output (they live in the GPU VO shaders). So colour grade routes through a lavfi eq/colorlevels filter in the same live vf chain, not set_adjust→equalizer-properties. Grade + enhance + key unify into one live chain. (--alpha option doesn't exist in this build and isn't needed.)

Two scope decisions taken up front (user, 2026-06-23):

  • Replace VLC outright. ferail-video-vlc is removed, not kept as a fallback: sequenced so mpv reaches frame-pull/seek/grade parity and passes its integration test first, with VLC deleted in the same phase. The viewer is never left without a working video path between phases.
  • N-layer stack. The compositor supports an arbitrary stack of keyed layers from the start (not a fixed two). The data model is N from day one; the performant ceiling of the CPU-buffer-pull path is a handful of layers at ≤1080p: see Performance.

Frame orientation (rotated video) - fixed 2026-06-25

Portrait phone clips (e.g. iPhone .MOV, rot=90) crashed libmpv's software render: Assertion failed: (x1 <= img->w && y1 <= img->h) in mp_image_crop. The libmpv VO advertises rotation support, so mpv's autorotate filter steps aside and leaves rotation to the VO, but the SW render path then computes the crop in rotated space (height 1280) and applies it to the un-rotated 1280×720 source, overflowing img->h. (It's an upstream mpv bug; not patchable here: libmpv is a runtime dlopen.)

Fix, entirely in ferail-video-mpv/src/imp.rs:

  • Set video-rotate=no so mpv never rotates: it hands us the native frame.
  • Read the intended rotation from video-dec-params/rotate, not video-params/rotate. The latter is the post-rotation value and video-rotate=no zeroes it; the decoder params keep the clockwise rotation regardless. (Verified: with video-rotate=no, video-params/rotate=0 but video-dec-params/rotate=90.)
  • Rotate the BGRA buffer ourselves (rotate_bgra, clockwise, dims swapped for 90/270). Direction matches mpv/ffmpeg (validated pixel-wise: see below).

rotate_bgra is per-frame and bounds-checked; cheap for a portrait preview, optimise only if a hot path needs it. Unit-tested for 90/180/270 geometry.

Diagnosing decode/render crashes (no GUI needed)

This class of bug lives in the frame-pull path (copy_frame), so it reproduces headlessly: no clicking required:

  • mpv's own log → stderr, opt-in: FERAIL_MPV_LOG=v (or debug). Off by default (error). The decoder/VO setup lines name the geometry, hwdec, and rotation: this is what revealed the rotation crash. (mpv_request_log_messages wired in imp.rs; a crash deep in libmpv leaves a breadcrumb either way.)
  • Headless probe drives the real backend against any file: cargo run -p ferail-video-mpv --example probe -- <video> [frames] (--raw out.bgra dumps the first frame for a visual/diff check).
  • Capture a crash backtrace non-interactively under lldb batch mode: lldb --batch -o run -k 'bt' -k 'quit' -- target/debug/examples/probe <video>
  • Rotated fixtures with ffmpeg 8 (the rotate= metadata tag is ignored; use the -display_rotation input option + stream copy): ffmpeg -f lavfi -i testsrc=d=1:s=1280x720 -pix_fmt yuv420p base.mp4 then ffmpeg -display_rotation 270 -i base.mp4 -c copy rot90.mp4 (mpv then reads rot=90). Diff our rotation against ffmpeg's own auto-rotate (ffmpeg -i rot90.mp4 -frames:v 1 ref.png) to confirm direction.

Platform tagging convention

Same as VIEWER.md: [mac] = macOS-only today; [win-parity] = the named Windows equivalent for later. libmpv is itself cross-platform, so most of this is platform-neutral; only the dylib-discovery and hardware-decode knobs are tagged.

Why mpv (the motivation, tied to the VLC code we have)

The libvlc backend works, but three properties of libvlc shaped real complexity that libmpv removes:

  1. libvlc can't change a video filter live. The whole seamless-reopen apparatus in the viewer: commit_video_enhance, video_pending_seek, video_repause, the discard-the-pre-seek-frame logic: exists only to work around that (video-filter is an instance arg; changing it means a new libvlc_new). mpv's vf chain is settable at runtime (mpv_command(["vf","set",…])). That apparatus deletes, and: the reason it matters here: live filters are the enabling primitive for a live transparent-colour picker (retune the key while watching it).
  2. libvlc has no reverse frame-step. VlcStream::step fakes backward steps by nudging the clock. mpv has frame-back-step.
  3. Same windowless pull model, near-zero trait churn. mpv's software render API (mpv_render_context_render into a CPU buffer) maps onto the existing copy_frame → tightly-packed BGRA seam exactly like libvlc's vmem did. ferail-video-mpv is a sibling of ferail-video-vlc; the VideoBackend/VideoStream trait barely moves.

Other parity: libmpv is mac/Win/Linux with the same runtime-dlopen model and the same LGPL "dynamic-link only" constraint as libvlc, so the cross-platform dynload module from the VLC crate is reusable nearly verbatim.

Honest caveats (these shape the plan)

  • No canonical bundled location like VLC.app/Contents/MacOS/lib. libmpv ships via Homebrew ($(brew --prefix mpv)/lib/libmpv.*.dylib) or inside mpv.app/Contents/Frameworks. The Settings path field must probe both and fail soft. [mac]
  • One genuine unknown: does mpv's software render emit a real alpha channel? Resolved in Phase 0: yes: keying lives in mpv's filter chain (see findings).

Existing anchors (verified 2026-06-23)

What Where
Video provider seam (VideoBackend/VideoStream, BGRA pull) crates/ferail-core/src/video.rs
VLC backend (hand-written FFI, runtime dlopen, cross-platform dynload) crates/ferail-video-vlc/src/imp.rs
Backend selection + path pref viewer/window.rs resolve_vlc_pref (:406); viewer/backend_native.rs video_backend()
Single-video frame pull (~60 Hz) viewer/window.rs start_video_poll (:1028), video_poll_tick (:1050)
Single-video render (one img via stage::layout) viewer/window.rs video_stage (:1944), stage_area (:1991)
Seamless-reopen apparatus (VLC-only; to delete) viewer/window.rs commit_video_enhance (:912), video_pending_seek use (:1063)
Per-frame CPU grade precedent viewer/window.rs graded_video (:1716), apply_color_adjust
GPUI already alpha-blends img per-pixel transparency (transparent PNGs over area.bg) viewer/loader.rs (RGBA→BGRA) + stage_area drawing img over bg
VLC feature wiring (mirror for mpv, then remove vlc) crates/ferail-gpui/Cargo.toml vlc feature + 3 target blocks
Spike precedent spikes/vlc-probe/ (standalone crate, dlopen FFI, skip-if-absent)

The last anchor is decisive for compositing: stacking N keyed videos is the same GPU-alpha-blend mechanism the viewer already uses to draw a transparent PNG over its canvas, not new rendering tech.

Trait changes (additive, native keeps defaults)

In ferail-core/src/video.rs:

/// A transparent-colour key applied to a layer: pixels within `similarity`
/// of `color` go transparent; `blend` softens the edge. `None` = no key.
pub struct ChromaKey {
    pub color: [u8; 3],     // target RGB
    pub similarity: f32,    // 0..1: how close counts as "the colour"
    pub blend: f32,         // 0..1: edge feather
}

pub trait VideoStream {
    // … existing …

    /// Apply/clear a transparent-colour key live. Returns true if the backend
    /// keyed natively (mpv: a live `colorkey` vf) so the viewer skips its CPU
    /// key; false (native player) leaves the CPU key in charge.
    fn set_chroma_key(&mut self, _key: Option<ChromaKey>) -> bool { false }

    /// Change enhancement filters live (mpv). Returns true if applied live;
    /// false (native) means the viewer must re-open to change them, which,
    /// once VLC is gone, no shipped backend needs, so the re-open path goes.
    fn set_enhance(&mut self, _enhance: VideoEnhance) -> bool { false }
}

VideoBackend::open keeps its signature; mpv ignores the baked-at-open enhance and applies it live via set_enhance instead.

Grade also goes through the live chain. Phase 0 found mpv's equalizer properties (brightness/contrast/…) don't affect the SW-render output, so the mpv backend implements set_adjust by composing a lavfi eq/colorlevels filter into the same live vf chain as enhance and key, returning true so the viewer skips its CPU grade. Grade + enhance + key are one live filter chain.

Architecture

New crate crates/ferail-video-mpv/: same shape as ferail-video-vlc:

ferail-video-mpv/
  Cargo.toml         , only ferail-core; libmpv loaded at runtime
  src/lib.rs         : backend(mpv_path) -> Option<Box<dyn VideoBackend>>
  src/imp.rs         : libmpv FFI: create/initialize/render-sw/observe/command;
                        reuses the dynload mac/win/linux pattern from the VLC crate

Viewer changes in crates/ferail-gpui/src/viewer/window.rs:

  • The single video_overlay: Option<(stream, path)> becomes layers: Vec<VideoLayer> (bottom→top), each owning its stream, poll epoch, Option<Arc<RenderImage>> frame + seq, Option<ChromaKey>, and a mute flag.
  • start_video_poll/video_poll_tick generalise to drive every layer; each layer pulls and supersedes its own RenderImage.
  • stage_area folds the layers into stacked absolute img children, all laid out through the same stage::layout against the shared StageState, so GPUI's GPU compositor alpha-blends them and zoom/pan/fit stay aligned.
  • A Composite section in the Adjustments popup: add/remove/reorder layers, a transparent-colour swatch + eyedropper (click swatch → click stage → sample the pixel from the cached top frame, read-only, no I/O), and Similarity / Blend sliders feeding the live key.
  • The VLC-only seamless-reopen machinery is deleted; enhancement sliders route to live set_enhance.

Data flow (≤200 words)

Opening a video resolves the active backend (native / mpv) once and opens the base layer's stream, as today. Each layer in the stack runs its own ~60 Hz poll that pulls the newest BGRA frame via copy_frame and supersedes a per-layer RenderImage. A layer's transparent colour is applied as a live mpv colorkey filter (set_chroma_key), so keyed pixels arrive with alpha = 0; if a backend can't key live, a CPU key_bgra pass in that layer's poll does it off the paint path. stage_area stacks the layers' absolute img elements bottom→top, each laid out through the same stage::layout against the shared StageState, so GPUI alpha-blends them on the GPU and zoom/pan/fit stay aligned across layers. The eyedropper samples one pixel from a cached frame (read-only) to set a key colour; similarity/blend sliders retune the live filter without a re-open. Render reads only cached frames and state; decode runs on mpv's threads, keying is live-filtered or polled off-paint, and stale frames drop by epoch: the prime directive holds.

Prime-directive compliance

  • Render reads only cached RenderImages + in-memory layer/stage state, no path resolution, stat, SQLite, or process spawn on any render/hover/scroll path.
  • Decode runs on mpv's own threads; keying is a live mpv filter (off our threads) or a CPU pass in the per-layer poll (foreground task, not the paint path), mirroring the existing graded_video/apply_color_adjust precedent.
  • Eyedropper sampling reads a cached BGRA buffer; no I/O.
  • Per-layer epochs drop stale/late frames, same idiom as today.

Performance (the honest ceiling)

The keying and compositing are nearly free; the cost that scales is the CPU-buffer pull, and N layers make it linear:

Stage Cost Verdict
Decode (N streams) mpv hwdec=videotoolbox [mac] / d3d11va [win-parity]: dedicated HW 2–3× 1080p30 trivial on Apple Silicon
Keying live mpv colorkey/chromakey lavfi filter, on mpv's threads free to us; never touches the main thread
Compositing GPUI GPU alpha-blend of stacked imgs free: the path transparent PNGs already use
SW readback + RenderImage upload, per layer per frame ~W·H·4 bytes × fps × N the real ceiling: comfy to ~2–3 layers @1080p; 4+ or any 4K layer competes with the UI

Mitigations (all cheap): only poll layers that are present; mute + optionally downscale lower layers; drop late frames by epoch; a soft layer-count guard that log()s when the stack exceeds what the CPU path serves well (no silent truncation).

The escalation, when measured need arrives: remove the readback entirely with GPU surfaces: gpui::surface(CVPixelBuffer) (already a deferred item). That makes N-layer 4K cheap but is a substantial rewrite of the frame path; per slow-AI "naive version first", it's a documented follow-up, not an MVP blocker. Phase 4 ships on the proven RenderImage-pull path and we escalate only if real layer counts demand it.

Phases

Each phase ends green (cargo check + cargo test), with a NOTES.md entry; UI phases add a screenshot under screenshots/.

  1. spikes/mpv-probe/ (throwaway): pull a SW-rendered frame; set a colour property live; change vf live; and the gate: does SW render emit real alpha from a colorkey filter? Resolves where keying runs. Delete spikes/ once the binding + alpha decision is recorded in NOTES.md.
  2. ferail-video-mpv to parity, then remove VLC: open/pull/seek/step/ grade behind the Plugins dropdown; integration test mirroring the VLC one; then delete ferail-video-vlc, the vlc feature, and the seamless-reopen apparatus in the same phase.
  3. Live enhance: denoise/sharpen/deband/grain via live set_enhance.
  4. Chroma key, single layer: ChromaKey state + similarity/blend + eyedropper; keyed holes show the stage background through. Eyedropper glyph added (check the spare Lucide pipette first) + ICONS.md.
  5. N-layer stack: Vec<VideoLayer>, per-layer poll + key, stacked compositing, soft layer-count guard, layer add/remove/reorder UI.
  6. Docs / icons / cleanup: finalise this doc, NOTES.md decision log, TODO follow-ups (incl. the GPU-surface escalation), ICONS.md.

Open questions / deferred

  • SW-render alpha (Phase 0 decides): resolved: alpha survives, key in the mpv filter chain; the CPU key_bgra fallback is unneeded.
  • GPU-surface compositing (gpui::surface) for high layer counts / 4K: the performance escalation, deferred until measured.
  • libmpv discovery defaults on Windows/Linux [win-parity]: mirror the mac Homebrew/mpv.app probe.
  • Per-layer audio policy beyond the per-window mute toggle (muted by default).
  • Layer sourcing UX (playlist neighbour vs file picker vs drag-in): settle in Phase 4.