[Bug]: H.264 recordings contain invalid access units and Preview is unavailable

[Arasshu]

Search existing issues

• I have searched the existing issues

Describe the bug

When recording the screen on Arch Linux using OpenScreen under a GNOME Wayland session, the recording is created successfully as an MP4 file, but the built-in Preview is unavailable.

The generated MP4 file contains a valid H.264 video stream and can be played/re-encoded with FFmpeg, but FFmpeg reports invalid H.264 access units when decoding the original recording:

[h264 @ ...] missing picture in access unit with size 241
[h264 @ ...] no frame!
[vist#0:0/h264 @ ...] [dec:h264 @ ...] Decoding error: Invalid data found when processing input

OpenScreen itself reports:

Avcodec_send_packet: -1094995529
Preview unavailable in this machine

The issue persists when launching OpenScreen with --disable-gpu.

Expected behavior

When recording the screen on Arch Linux using OpenScreen under a GNOME Wayland session, the recording is created successfully as an MP4 file, but the built-in Preview is unavailable.

The generated MP4 file contains a valid H.264 video stream and can be played/re-encoded with FFmpeg, but FFmpeg reports invalid H.264 access units when decoding the original recording:

[h264 @ ...] missing picture in access unit with size 241
[h264 @ ...] no frame!
[vist#0:0/h264 @ ...] [dec:h264 @ ...] Decoding error: Invalid data found when processing input

OpenScreen itself reports:

Avcodec_send_packet: -1094995529
Preview unavailable in this machine

The issue persists when launching OpenScreen with --disable-gpu.

To Reproduce

1. Start a GNOME Wayland session on Arch Linux.

2. Launch OpenScreen.

3. Start a screen recording.

4. Stop the recording after a short period.

5. Open the generated recording in OpenScreen.

6. Observe that the Preview is unavailable.

7. Check the generated MP4 with FFmpeg:

   ffmpeg -v error -i recording.mp4 -f null -

8. FFmpeg reports H.264 decoding errors such as:

   missing picture in access unit with size 241

   no frame!

   Decoding error: Invalid data found when processing input

Screenshots

OS

Linux

OS Version

Arch Linux 7.2.2-arch1-1

Other OS

No response

Browser

Other

Browser Version

Brave Browser 152.1.94.117

Other Browser

Brave

Device Type

Laptop

Other Device

No response

Additional context

System

• OS: Arch Linux
• Desktop: GNOME
• Session: Wayland
• GPU: AMD Radeon HD 8400 / R3 Series
• GPU architecture: Kabini
• Kernel: 7.2.2-arch1-1
• Mesa: 26.2.2-arch1.1
• VA-API: libva 2.24.0
• OpenScreen: [version]

PipeWire

PipeWire, pipewire-pulse and WirePlumber are running normally.

VA-API

vainfo reports H.264 High profile support:

• H264 High VLD
• H264 High EncSlice

Generated recording

The generated MP4 is approximately:

• Duration: 49.68 seconds
• Video: H.264 High
• Resolution: 1366x736
• Framerate: ~60 FPS
• Bitrate: ~4.9 Mbps

The MP4 is not empty and can be re-encoded successfully with FFmpeg. However, decoding the original file directly produces H.264 errors:

missing picture in access unit with size 241
no frame!
Decoding error: Invalid data found when processing input

Launching OpenScreen with:

openscreen --disable-gpu

does not resolve the Preview problem.

This suggests that the recording pipeline is producing an MP4 which is mostly valid/playable but contains at least one malformed H.264 access unit, and OpenScreen's preview decoder fails when processing it.

FFmpeg

ffmpeg -hwaccels reports VA-API support, and vainfo confirms H.264 encoding/decoding capabilities on the AMD GPU.

I have not yet determined whether the malformed H.264 packet is produced by the capture pipeline, encoder, or muxing stage.