SteamRT3 Gentoo/AMD Radeon: streaming/recording does not work.

[mlauss2]

Your system information

• Steam client version (build number or date): 1787790984, SteamRT3
• Distribution (e.g. Ubuntu): Gentoo
• Opted into Steam client beta?: [Yes/No] Yes
• Have you checked for system updates?: [Yes/No] Yes
• Steam Logs: [generate by running this command in a terminal tar -zcvf ~/Desktop/steam-logs.tar.gz ~/.steam/steam/logs] Attached.
• GPU: Nvidia/Intel/AMD AMD 7900XTX

Please describe your issue in as much detail as possible:

Recording/Streaming does not work, it looks like the SteamRT3 client has problems with VAAPI:

steam-logs.tar.gz

[2026-08-28 12:58:21][28.283832] >>> Switching video stream from NONE to GameOverlay_MovieStream_15883
[2026-08-28 12:58:21][28.301554] Building new pipeline
[2026-08-28 12:58:21][28.301572] Trying to create an encoder for recording: [hardware_enabled=true][hardware_suppressed=false]
[2026-08-28 12:58:21][28.301579] Allowed Codecs: 5,4
[2026-08-28 12:58:21][28.302120] Created encoder VAAPI for codec 5
[2026-08-28 12:58:21][28.302742] CGameStreamVideoStageVAAPI: Reinitializing 2580x1080 at 24000 Kbit/s and 60.00 FPS with quality 1
[2026-08-28 12:58:21][28.303660] libav: Failed to initialise VAAPI connection: -1 (unknown libva error).
[2026-08-28 12:58:21][28.303786] CGameStreamVideoStageVAAPI: Failed to create device context: Input/output error
[2026-08-28 12:58:21][28.303789] Failed to pre-process frame
[2026-08-28 12:58:21][28.322791] Encoder stage failed [type=5, codec=5]
[2026-08-28 12:58:21][28.322834] Building new pipeline
[2026-08-28 12:58:21][28.322840] Trying to create an encoder for recording: [hardware_enabled=true][hardware_suppressed=false]
[2026-08-28 12:58:21][28.322844] Allowed Codecs: 5,4
[2026-08-28 12:58:21][28.322848] Failed encoders: 5/5
[2026-08-28 12:58:21][28.323219] Created encoder VAAPI for codec 4
[2026-08-28 12:58:21][28.323394] CGameStreamVideoStageVAAPI: Reinitializing 2580x1080 at 24000 Kbit/s and 60.00 FPS with quality 1
[2026-08-28 12:58:21][28.323578] libav: Failed to initialise VAAPI connection: -1 (unknown libva error).
[2026-08-28 12:58:21][28.323669] CGameStreamVideoStageVAAPI: Failed to create device context: Input/output error
[2026-08-28 12:58:21][28.323672] Failed to pre-process frame
[2026-08-28 12:58:21][28.333964] Encoder stage failed [type=5, codec=4]
[2026-08-28 12:58:21][28.333998] Building new pipeline
[2026-08-28 12:58:21][28.334004] Trying to create an encoder for recording: [hardware_enabled=true][hardware_suppressed=false]
[2026-08-28 12:58:21][28.334013] Allowed Codecs: 5,4
[2026-08-28 12:58:21][28.334018] Failed encoders: 5/5, 5/4
[2026-08-28 12:58:21][28.334516] Couldn't load libx264.so.142, x264 support disabled
[2026-08-28 12:58:21][28.371294] Failed to create an encoder
[2026-08-28 12:58:21][28.408648] Building new pipeline
[2026-08-28 12:58:21][28.408667] Trying to create an encoder for recording: [hardware_enabled=true][hardware_suppressed=false]
[2026-08-28 12:58:21][28.408674] Allowed Codecs: 5,4
[2026-08-28 12:58:21][28.408678] Failed encoders: 5/5, 5/4
[2026-08-28 12:58:21][28.409116] Couldn't load libx264.so.142, x264 support disabled
[2026-08-28 12:58:21][28.409120] Failed to create an encoder
[...]

VAAPI on the host does work:
$ vainfo
Trying display: wayland
libva info: VA-API version 1.25.0
libva info: User environment variable requested driver 'radeonsi'
libva info: Trying to open /usr/lib64/va/drivers/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_1_25
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.25 (libva 2.25.0.1)
vainfo: Driver version: Mesa Gallium driver 26.3.0-devel for AMD Radeon RX 7900 XTX (radeonsi, navi31, ACO, DRM 3.64, 7.2.0-15794-g1b78070aaef6-dirty)
vainfo: Supported profile and entrypoints
VAProfileH264ConstrainedBaseline: VAEntrypointVLD
VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
VAProfileH264Main : VAEntrypointVLD
VAProfileH264Main : VAEntrypointEncSlice
VAProfileH264High : VAEntrypointVLD
VAProfileH264High : VAEntrypointEncSlice
VAProfileHEVCMain : VAEntrypointVLD
VAProfileHEVCMain : VAEntrypointEncSlice
VAProfileHEVCMain10 : VAEntrypointVLD
VAProfileHEVCMain10 : VAEntrypointEncSlice
VAProfileJPEGBaseline : VAEntrypointVLD
VAProfileVP9Profile0 : VAEntrypointVLD
VAProfileVP9Profile2 : VAEntrypointVLD
VAProfileAV1Profile0 : VAEntrypointVLD
VAProfileAV1Profile0 : VAEntrypointEncSlice
VAProfileAV1Profile2 : VAEntrypointVLD
VAProfileNone : VAEntrypointVideoProc

Steps for reproducing this issue:

1. Have SteamRT3 client on latest gentoo linux with recent amd radeon hardware and up-to-date mesa and vaapi
2. enable game recording in steam client
3. watch it fail to record anything (or stream to steam link app/hardware)

steam-logs.tar.gz