The optional video backend is libmpv, dlopened at runtime, and on top of it a transparent-colour (chroma-key) stage: the user picks a colour on a video and sees through it to whatever is beneath, including other videos that are themselves keyed. Layers are separate transparent viewer windows, stacked and composited by the window server rather than inside the app.
← Feature notes index · Viewer · Architecture · TODO
- What is built
- Frame orientation (rotated video) - fixed 2026-06-25
- Diagnosing decode/render crashes (no GUI needed)
- Platform tagging convention
- Why mpv (the motivation, tied to the VLC code we have)
- Existing anchors (verified 2026-06-23)
- Trait changes (additive, native keeps defaults)
- Architecture
- Performance (the honest ceiling)
- Phases
- Open questions / deferred
Phases 0–4 shipped (2026-06-23), all compile-green; see commits + the NOTES.md decision log. At a glance:
- 0 spike: SW render emits real alpha ✅
- 1a
ferail-video-mpvcrate to parity ✅ - 1b optional backend swapped to mpv,
ferail-video-vlcdeleted ✅ - 2 live
set_enhance, VLC-era reopen apparatus removed ✅ - 3 single-layer chroma key + eyedropper ✅ (screenshot-verified)
- UI professional popup rework (labeled sections, swatch/hex/Pick) ✅
- fix
vo=libmpv: mpv was opening its own native window + deadlocking ✅ - 4 transparent stacking windows (replaced the in-app layer stack) ✅
- 5/4b: deferred items below (rename: done)
Compositing model: transparent stacking windows (2026-06-23). The first
Phase 4 cut composited background videos inside one window (a Vec<BgLayer>
stack with a "Layers" popup section). That was the wrong model, per user
direction, the real design is one keyed video per window, and the windows
themselves are transparent, so independent viewer windows stack and the
OS window server composites them on the GPU. So the in-app layer stack was
removed and replaced by a Transparent toggle in the viewer toolbar
(window.set_background_appearance(Transparent) + transparent root/stage
backgrounds). With it on, the chroma-keyed-transparent video pixels and the
window background are see-through, so stacking two viewer windows (pair with
Stay on top) shows the lower video through the upper's keyed regions:
composited by macOS, no app-side compositing. This also answers "rendering on
the GPU": layer compositing is now fully GPU via the window server. The
remaining CPU cost is the per-window SW frame pull; eliminating that (mpv GL
render → IOSurface → gpui::surface) is the documented escalation, still open.
The earlier deadlock fix: without an explicit vo, mpv created its native
macOS gpu (Cocoa) video output: a separate "- mpv" window with a
CVDisplayLink that dispatch_syncs to the main thread, deadlocking on
teardown. vo=libmpv makes the SW render context the only output (no window).
Phase 3: single-layer chroma key: a "Transparent colour" section
in the adjustments popup (mpv video only) with an on/off toggle, an
eyedropper swatch (click it, then click the video to sample the key
colour from the live frame), and Similarity (range width) + Blend
(edge feather) sliders. The key is pushed live via set_chroma_key, so keyed
pixels arrive transparent and the stage background shows through. No new icon
(the swatch is the eyedropper affordance), so ICONS.md is untouched. cargo check -p ferail-gpui --features mpv is green.
Verified by screenshot + UI reworked. The adjustments popup renders correctly with the full mpv-video control set (a new screenshot-only
--viewer-adjust-videofixture forces the panel without a live stream, since the real frame-pull poll never lets a headless render settle). The popup was then reworked to look more professional: labeled Colour / Enhance / Transparent colour sections (replacing bare separators), and the key picker now shows a swatch + hex readout + a clear Pick button (arms the eyedropper) instead of a tiny "click swatch" hint. Still pending: a screenshot of a live keyed video (the keyed pixels actually transparent), which needs the poll to run: a follow-up.
Per-window mute shipped, each viewer window has a mute toggle in the video
transport (volume-x/volume-2 button → set_muted), muted by default,
so stacked transparent viewers never all play audio at once; sound is opt-in
per window. (An earlier focus-follows-audio version: auto-mute while the
window wasn't active: was replaced by the explicit toggle.)
The idea was to skip the per-frame CPU readback/upload by handing mpv's frame
to gpui's surface element (gpui::surface(CVPixelBuffer), GPU-composited via
Metal). The spike killed it for our case:
- gpui's surface fragment shader (
gpui_macos/.../shaders.metal,surface_fragment) is hardcoded for opaque bi-planar YUV (NV12): it samples a Y plane + a CbCr plane, does YCbCr→RGB, and returns alpha = 1.0. So a surface cannot carry the chroma-key transparency: the keyed alpha would be discarded, breaking the transparent-window feature (the whole point). - gpui is a pinned git dep with no
[patch]/fork; this repo forks small pieces locally rather than the engine, so patching the surface shader to accept BGRA-with-alpha isn't on the table.
Conclusion: the SW RenderImage path stays: it carries alpha, so it's the
only path that supports keyed/transparent video, which is the headline feature.
The per-frame upload is the accepted cost. Note the layer compositing is
already fully on the GPU (the OS window server composites the stacked
transparent windows), so the only remaining CPU work is each window's frame
rasterize+upload. A future opaque-only fast path via gpui::surface (YUV, no
key) is possible if non-keyed playback ever becomes a bottleneck, but it'd be a
second code path for marginal gain.
Phase 2 recap: removed the VLC-era seamless-reopen apparatus:
commit_video_enhance now pushes filters through live set_enhance; the
video_pending_seek/video_repause deferral and the poll's pre-seek-frame
dance are gone (−67 lines in window.rs). The decision log is in
NOTES.md (2026-06-23 entry).
Every user-facing vlc identifier was renamed to mpv:
- the cargo feature
vlc→mpv(build with--features mpv), - the persisted
video_backendvalue"vlc"→"mpv"and fieldvlc_app_path→mpv_path(the Settings "mpv library" input), - the Settings → Plugins dropdown label "VLC" → "mpv", plus all the
internal symbols (
resolve_mpv_pref,default_mpv_path,MPV_VIDEO_EXTS, …) and comments.
Back-compat: app_state still accepts a legacy video_backend=vlc on load
and normalises it to mpv, so existing configs keep selecting the provider; the
next settings save writes mpv.
- SW render → BGRA pull works, same shape as libvlc's vmem, so the
copy_frame → (w,h,BGRA)seam is untouched. - THE GATE: SW render emits a real alpha channel. PASS. A live
colorkeyfilter produced correct per-pixel alpha through SW render (keyed background transparent, foreground opaque:screenshots/mpv-probe-B_alpha_green.png). Keying lives in mpv's filter chain, live and off our threads, not a CPU pass. Recipe: end the vf chain in an alpha format + requestbgra:vf = lavfi=[…,format=rgba,colorkey=color=0xRRGGBB:similarity=…:blend=…]. - Live
vfchange applies with no re-open → the VLC reopen apparatus goes. - Correction: mpv's
brightness/contrast/… equalizer properties are a no-op on the SW-render output (they live in the GPU VO shaders). So colour grade routes through a lavfieq/colorlevelsfilter in the same live vf chain, notset_adjust→equalizer-properties. Grade + enhance + key unify into one live chain. (--alphaoption doesn't exist in this build and isn't needed.)
Two scope decisions taken up front (user, 2026-06-23):
- Replace VLC outright.
ferail-video-vlcis removed, not kept as a fallback: sequenced so mpv reaches frame-pull/seek/grade parity and passes its integration test first, with VLC deleted in the same phase. The viewer is never left without a working video path between phases. - N-layer stack. The compositor supports an arbitrary stack of keyed layers from the start (not a fixed two). The data model is N from day one; the performant ceiling of the CPU-buffer-pull path is a handful of layers at ≤1080p: see Performance.
Portrait phone clips (e.g. iPhone .MOV, rot=90) crashed libmpv's
software render: Assertion failed: (x1 <= img->w && y1 <= img->h) in
mp_image_crop. The libmpv VO advertises rotation support, so mpv's
autorotate filter steps aside and leaves rotation to the VO, but the SW
render path then computes the crop in rotated space (height 1280) and applies
it to the un-rotated 1280×720 source, overflowing img->h. (It's an upstream
mpv bug; not patchable here: libmpv is a runtime dlopen.)
Fix, entirely in ferail-video-mpv/src/imp.rs:
- Set
video-rotate=noso mpv never rotates: it hands us the native frame. - Read the intended rotation from
video-dec-params/rotate, notvideo-params/rotate. The latter is the post-rotation value andvideo-rotate=nozeroes it; the decoder params keep the clockwise rotation regardless. (Verified: withvideo-rotate=no,video-params/rotate=0 butvideo-dec-params/rotate=90.) - Rotate the BGRA buffer ourselves (
rotate_bgra, clockwise, dims swapped for 90/270). Direction matches mpv/ffmpeg (validated pixel-wise: see below).
rotate_bgra is per-frame and bounds-checked; cheap for a portrait preview,
optimise only if a hot path needs it. Unit-tested for 90/180/270 geometry.
This class of bug lives in the frame-pull path (copy_frame), so it reproduces
headlessly: no clicking required:
- mpv's own log → stderr, opt-in:
FERAIL_MPV_LOG=v(ordebug). Off by default (error). The decoder/VO setup lines name the geometry, hwdec, and rotation: this is what revealed the rotation crash. (mpv_request_log_messageswired inimp.rs; a crash deep in libmpv leaves a breadcrumb either way.) - Headless probe drives the real backend against any file:
cargo run -p ferail-video-mpv --example probe -- <video> [frames](--raw out.bgradumps the first frame for a visual/diff check). - Capture a crash backtrace non-interactively under lldb batch mode:
lldb --batch -o run -k 'bt' -k 'quit' -- target/debug/examples/probe <video> - Rotated fixtures with ffmpeg 8 (the
rotate=metadata tag is ignored; use the-display_rotationinput option + stream copy):ffmpeg -f lavfi -i testsrc=d=1:s=1280x720 -pix_fmt yuv420p base.mp4thenffmpeg -display_rotation 270 -i base.mp4 -c copy rot90.mp4(mpv then readsrot=90). Diff our rotation against ffmpeg's own auto-rotate (ffmpeg -i rot90.mp4 -frames:v 1 ref.png) to confirm direction.
Same as VIEWER.md: [mac] = macOS-only today; [win-parity] = the named Windows equivalent for later. libmpv is itself cross-platform, so most of this is platform-neutral; only the dylib-discovery and hardware-decode knobs are tagged.
The libvlc backend works, but three properties of libvlc shaped real complexity that libmpv removes:
- libvlc can't change a video filter live. The whole seamless-reopen
apparatus in the viewer:
commit_video_enhance,video_pending_seek,video_repause, the discard-the-pre-seek-frame logic: exists only to work around that (video-filteris an instance arg; changing it means a newlibvlc_new). mpv'svfchain is settable at runtime (mpv_command(["vf","set",…])). That apparatus deletes, and: the reason it matters here: live filters are the enabling primitive for a live transparent-colour picker (retune the key while watching it). - libvlc has no reverse frame-step.
VlcStream::stepfakes backward steps by nudging the clock. mpv hasframe-back-step. - Same windowless pull model, near-zero trait churn. mpv's software
render API (
mpv_render_context_renderinto a CPU buffer) maps onto the existingcopy_frame → tightly-packed BGRAseam exactly like libvlc's vmem did.ferail-video-mpvis a sibling offerail-video-vlc; theVideoBackend/VideoStreamtrait barely moves.
Other parity: libmpv is mac/Win/Linux with the same runtime-dlopen model and
the same LGPL "dynamic-link only" constraint as libvlc, so the cross-platform
dynload module from the VLC crate is reusable nearly verbatim.
- No canonical bundled location like
VLC.app/Contents/MacOS/lib. libmpv ships via Homebrew ($(brew --prefix mpv)/lib/libmpv.*.dylib) or insidempv.app/Contents/Frameworks. The Settings path field must probe both and fail soft. [mac] One genuine unknown: does mpv's software render emit a real alpha channel?Resolved in Phase 0: yes: keying lives in mpv's filter chain (see findings).
| What | Where |
|---|---|
Video provider seam (VideoBackend/VideoStream, BGRA pull) |
crates/ferail-core/src/video.rs |
VLC backend (hand-written FFI, runtime dlopen, cross-platform dynload) |
crates/ferail-video-vlc/src/imp.rs |
| Backend selection + path pref | viewer/window.rs resolve_vlc_pref (:406); viewer/backend_native.rs video_backend() |
| Single-video frame pull (~60 Hz) | viewer/window.rs start_video_poll (:1028), video_poll_tick (:1050) |
Single-video render (one img via stage::layout) |
viewer/window.rs video_stage (:1944), stage_area (:1991) |
| Seamless-reopen apparatus (VLC-only; to delete) | viewer/window.rs commit_video_enhance (:912), video_pending_seek use (:1063) |
| Per-frame CPU grade precedent | viewer/window.rs graded_video (:1716), apply_color_adjust |
GPUI already alpha-blends img per-pixel transparency (transparent PNGs over area.bg) |
viewer/loader.rs (RGBA→BGRA) + stage_area drawing img over bg |
VLC feature wiring (mirror for mpv, then remove vlc) |
crates/ferail-gpui/Cargo.toml vlc feature + 3 target blocks |
| Spike precedent | spikes/vlc-probe/ (standalone crate, dlopen FFI, skip-if-absent) |
The last anchor is decisive for compositing: stacking N keyed videos is the same GPU-alpha-blend mechanism the viewer already uses to draw a transparent PNG over its canvas, not new rendering tech.
In ferail-core/src/video.rs:
/// A transparent-colour key applied to a layer: pixels within `similarity`
/// of `color` go transparent; `blend` softens the edge. `None` = no key.
pub struct ChromaKey {
pub color: [u8; 3], // target RGB
pub similarity: f32, // 0..1: how close counts as "the colour"
pub blend: f32, // 0..1: edge feather
}
pub trait VideoStream {
// … existing …
/// Apply/clear a transparent-colour key live. Returns true if the backend
/// keyed natively (mpv: a live `colorkey` vf) so the viewer skips its CPU
/// key; false (native player) leaves the CPU key in charge.
fn set_chroma_key(&mut self, _key: Option<ChromaKey>) -> bool { false }
/// Change enhancement filters live (mpv). Returns true if applied live;
/// false (native) means the viewer must re-open to change them, which,
/// once VLC is gone, no shipped backend needs, so the re-open path goes.
fn set_enhance(&mut self, _enhance: VideoEnhance) -> bool { false }
}VideoBackend::open keeps its signature; mpv ignores the baked-at-open
enhance and applies it live via set_enhance instead.
Grade also goes through the live chain. Phase 0 found mpv's equalizer
properties (brightness/contrast/…) don't affect the SW-render output, so
the mpv backend implements set_adjust by composing a lavfi
eq/colorlevels filter into the same live vf chain as enhance and key,
returning true so the viewer skips its CPU grade. Grade + enhance + key are
one live filter chain.
New crate crates/ferail-video-mpv/: same shape as ferail-video-vlc:
ferail-video-mpv/
Cargo.toml , only ferail-core; libmpv loaded at runtime
src/lib.rs : backend(mpv_path) -> Option<Box<dyn VideoBackend>>
src/imp.rs : libmpv FFI: create/initialize/render-sw/observe/command;
reuses the dynload mac/win/linux pattern from the VLC crate
Viewer changes in crates/ferail-gpui/src/viewer/window.rs:
- The single
video_overlay: Option<(stream, path)>becomeslayers: Vec<VideoLayer>(bottom→top), each owning its stream, poll epoch,Option<Arc<RenderImage>>frame + seq,Option<ChromaKey>, and a mute flag. start_video_poll/video_poll_tickgeneralise to drive every layer; each layer pulls and supersedes its ownRenderImage.stage_areafolds the layers into stacked absoluteimgchildren, all laid out through the samestage::layoutagainst the sharedStageState, so GPUI's GPU compositor alpha-blends them and zoom/pan/fit stay aligned.- A
Compositesection in the Adjustments popup: add/remove/reorder layers, a transparent-colour swatch + eyedropper (click swatch → click stage → sample the pixel from the cached top frame, read-only, no I/O), and Similarity / Blend sliders feeding the live key. - The VLC-only seamless-reopen machinery is deleted; enhancement sliders route
to live
set_enhance.
Opening a video resolves the active backend (native / mpv) once and opens the
base layer's stream, as today. Each layer in the stack runs its own ~60 Hz
poll that pulls the newest BGRA frame via copy_frame and supersedes a
per-layer RenderImage. A layer's transparent colour is applied as a live
mpv colorkey filter (set_chroma_key), so keyed pixels arrive with
alpha = 0; if a backend can't key live, a CPU key_bgra pass in that layer's
poll does it off the paint path. stage_area stacks the layers' absolute
img elements bottom→top, each laid out through the same stage::layout
against the shared StageState, so GPUI alpha-blends them on the GPU and
zoom/pan/fit stay aligned across layers. The eyedropper samples one pixel from
a cached frame (read-only) to set a key colour; similarity/blend sliders
retune the live filter without a re-open. Render reads only cached frames and
state; decode runs on mpv's threads, keying is live-filtered or polled
off-paint, and stale frames drop by epoch: the prime directive holds.
- Render reads only cached
RenderImages + in-memory layer/stage state, no path resolution, stat, SQLite, or process spawn on any render/hover/scroll path. - Decode runs on mpv's own threads; keying is a live mpv filter (off our
threads) or a CPU pass in the per-layer poll (foreground task, not the
paint path), mirroring the existing
graded_video/apply_color_adjustprecedent. - Eyedropper sampling reads a cached BGRA buffer; no I/O.
- Per-layer epochs drop stale/late frames, same idiom as today.
The keying and compositing are nearly free; the cost that scales is the CPU-buffer pull, and N layers make it linear:
| Stage | Cost | Verdict |
|---|---|---|
| Decode (N streams) | mpv hwdec=videotoolbox [mac] / d3d11va [win-parity]: dedicated HW |
2–3× 1080p30 trivial on Apple Silicon |
| Keying | live mpv colorkey/chromakey lavfi filter, on mpv's threads |
free to us; never touches the main thread |
| Compositing | GPUI GPU alpha-blend of stacked imgs |
free: the path transparent PNGs already use |
SW readback + RenderImage upload, per layer per frame |
~W·H·4 bytes × fps × N | the real ceiling: comfy to ~2–3 layers @1080p; 4+ or any 4K layer competes with the UI |
Mitigations (all cheap): only poll layers that are present; mute +
optionally downscale lower layers; drop late frames by epoch; a soft
layer-count guard that log()s when the stack exceeds what the CPU path
serves well (no silent truncation).
The escalation, when measured need arrives: remove the readback entirely
with GPU surfaces: gpui::surface(CVPixelBuffer) (already a deferred item).
That makes N-layer 4K cheap but is a substantial rewrite of the frame path;
per slow-AI "naive version first", it's a documented follow-up, not an MVP
blocker. Phase 4 ships on the proven RenderImage-pull path and we escalate only
if real layer counts demand it.
Each phase ends green (cargo check + cargo test), with a NOTES.md entry;
UI phases add a screenshot under screenshots/.
spikes/mpv-probe/(throwaway): pull a SW-rendered frame; set a colour property live; changevflive; and the gate: does SW render emit real alpha from acolorkeyfilter? Resolves where keying runs. Deletespikes/once the binding + alpha decision is recorded in NOTES.md.ferail-video-mpvto parity, then remove VLC: open/pull/seek/step/ grade behind the Plugins dropdown; integration test mirroring the VLC one; then deleteferail-video-vlc, thevlcfeature, and the seamless-reopen apparatus in the same phase.- Live enhance: denoise/sharpen/deband/grain via live
set_enhance. - Chroma key, single layer:
ChromaKeystate + similarity/blend + eyedropper; keyed holes show the stage background through. Eyedropper glyph added (check the spare Lucidepipettefirst) + ICONS.md. - N-layer stack:
Vec<VideoLayer>, per-layer poll + key, stacked compositing, soft layer-count guard, layer add/remove/reorder UI. - Docs / icons / cleanup: finalise this doc, NOTES.md decision log, TODO follow-ups (incl. the GPU-surface escalation), ICONS.md.
SW-render alpha (Phase 0 decides): resolved: alpha survives, key in the mpv filter chain; the CPUkey_bgrafallback is unneeded.- GPU-surface compositing (
gpui::surface) for high layer counts / 4K: the performance escalation, deferred until measured. - libmpv discovery defaults on Windows/Linux [win-parity]: mirror the
mac Homebrew/
mpv.appprobe. - Per-layer audio policy beyond the per-window mute toggle (muted by default).
- Layer sourcing UX (playlist neighbour vs file picker vs drag-in): settle in Phase 4.