Drain texture cache while capture is stopped (#26)

[aldoborrero]

Summary

While capture is stopped, GPU memory imported from nested clients grows without bound and is only released on the next start_capture(). This fixes it by draining Smithay's deferred texture-destruction queue on each idle render tick.

Fixes #26.

Root cause

As diagnosed by @lonk42 in #26:

• The calloop render tick early-outs at src/lib.rs while !state.is_capturing && state.pending_screenshot.is_none(), so no Frame is ever finished.
• DmabufHandler::dmabuf_imported keeps importing client buffers while stopped — it is protocol-driven, not tick-driven.
• Smithay only issues glDeleteTextures / eglDestroyImageKHR from GlesRenderer::cleanup(), whose callers are finishing a Frame and Renderer::cleanup_texture_cache(). The latter was never called while capture was stopped, so the deferred-destruction queue was never drained.

The result: for as long as nested clients keep creating/destroying buffers with capture stopped, imported GPU memory accumulates. On NVIDIA it shows up as type G device memory (the reporter's climb: 167 → 5747 MiB); on an integrated GPU the buffers are system RAM, so it surfaces as RSS growth and eventual OOM.

Fix

Call Renderer::cleanup_texture_cache() on the renderer inside the idle early-out, so the deferred queue is drained every tick (~16 ms) while capture is stopped:

if !state.is_capturing && state.pending_screenshot.is_none() {
    // drain Smithay's deferred texture-destruction queue while stopped
    if let Some(renderer) = state.gles_renderer.as_mut() {
        let _ = renderer.cleanup_texture_cache();
    }
    return TimeoutAction::ToDuration(Duration::from_millis(16));
}

This matches the behaviour the reporter observed with the screenshot action, which forces a single render tick and drops memory back to baseline — cleanup_texture_cache() is that same drain, run unconditionally while idle. The Renderer trait is added to the existing smithay::backend::renderer import to bring the method into scope.

Validation

Built from source against the pinned smithay rev (ca932e04…) — compiles clean — and packaged/deployed in a selkies-based Wayland capture setup on an AMD 780M iGPU (h264_vaapi, PIXELFLUX_WAYLAND=true), where the pre-built wheel had been growing RSS until OOM. Capture start/stop is driven by browser clients connecting/disconnecting, exactly as in #26.

---

Disclosure: this patch was prepared with AI assistance. The diagnosis, reproduction, and the pointer to cleanup_texture_cache() are @lonk42's from #26.

[thelamer]

thanks for this PR, just so you are aware the next release of pixelflux will be here https://github.com/selkies-project/pixelflux. This transition is expected to occur in the next couple weeks and upstream selkies will be taking over all aspects of the project. Until then stuff on this side is kind of frozen in time.