[MaxMB15]

What this changes

Step 3 of AirPlay. When an AirPlay receiver can't play a stream as it is, the iPhone or iPad converts it and serves it as HLS on the Wi-Fi network, and the TV plays that. This covers most M3U live channels (raw MPEG-TS), MKV VOD, and MP2, AC-3, E-AC-3 or DTS audio.

This builds on cast's converter branch (feat/cast-converter, with cast's LanServer). It hasn't merged, so the diff also shows its changes. #323 and #320 are in dev.

• crates/hls-remux (new): a small C file and its Rust wrapper.
  • It opens the source with libavformat and copies the video.
  • It copies or converts one audio track. A track the receiver plays (AAC, MP3) is copied. Anything else becomes stereo AAC at 256 kb/s, with the AC-3, E-AC-3 and DTS decoders mixing down with the stream's own levels (downmix=stereo).
  • It cuts MPEG-TS segments in memory at video keyframes. Each segment starts with the PAT and SDT, so it plays on its own.
  • Why C: libav's structs change layout between FFmpeg versions, so the C is built against the real headers rather than declared by hand in Rust.
  • Why its own HLS: MPVKit's FFmpeg has no HLS muxer. main.swift in MPVKit 1.1.0-n9.0.2 turns off every muxer and enables only flac, dash, hevc, m4v, matroska, mov, mp4, mpegts and webm.
  • iOS: Xcode compiles the C in the app target against MPVKit's FFmpeg. That's the project.yml source ../../../../../crates/hls-remux/native, and a group plus a build file added to project.pbxproj by hand, since there's no XcodeGen here.
  • Linux: the crate's build.rs builds the C against the system FFmpeg, only for tests. The app takes the crate as a Linux dev-dependency, so the .deb, .rpm and AppImage never link FFmpeg.
  • No aac_at: MPVKit's build turns off every encoder except ffmpeg's own aac and a few others.
• mvp_core::convert::hls (new): the media and master playlists, the CODECS names (avc1.* from avcC or Annex B, mp4a.40.2, mp4a.40.34), and a Pacer.
• src-tauri/src/hls_out.rs (new): one conversion.
  • Serving: on cast's LanServer::start_for(Peers::Subnet), at the iPhone's Wi-Fi address (en0, from getifaddrs in wifi.rs). It answers only that subnet, and only under a random path.
  • Memory: a remux runs far faster than realtime. So the converter keeps 15 segments, and the pacer holds it 20 s of media ahead of the receiver's playback clock. The clock stops while AVPlayer is paused, which the app hears from airplay://state.
  • Startup: it returns once three segments are up, as Apple asks for live HLS. A refused login fails before the TV hears anything.
  • Audio choice: cast's pick_audio with AirPlay's codec list.
  • Seeking: a seek on a converted movie converts again from there.
• Commands: airplay_load_converted, airplay_end_conversion and airplay_converter_support (convert on iOS, no hardware encoder yet). The plugin counts a converted stream's position from where it started, and takes its length from the source.
• Frontend:
  • AirPlayDestination takes a sender profile and loads converted routes. A converted live channel can't pause, and a converted stream keeps one audio track. The capabilities go out with the state that brings them, as the fixed feat(player): send the player's controls through a playback destination #306 asks.
  • The AirPlay button asks before converting: "This iPhone converts the stream and sends it to the TV. Keep it on Wi-Fi and charged. It can lock."
• CI: mutants.yml and changed-code.yml install FFmpeg's dev packages for hls-remux's C, and install-linux-deps adds libswresample-dev. hls-remux has a place in the weekly mutants matrix. changed-code.py reads the workspace members, so the pull request shards pick the crate up by themselves. The iOS-only airplay/ios.rs and wifi.rs's interfaces are left out of cargo-mutants.
• Cast's files: one line in crates/core/src/convert/mod.rs (pub mod hls;). The test helper in crates/core/src/convert/tests.rs gets the new layout field, the same way cast's newer branch has it.

Not in this step: a video transcode with h264_videotoolbox (HEVC, MPEG-2 and interlaced still say they can't play), a WebVTT subtitle rendition, and choosing the audio language.

How it was tested

On Linux, with no display and no device. iOS can't be built here, so CI's iOS build is the first compile of the C against MPVKit and of the iOS Rust.

• hls-remux: 14 tests against the system FFmpeg 7.1. They use WAV and Y4M media made in the tests, from files and loopback servers, and read the output back through the same API. They cover:
  • AAC conversion and audio copy
  • keyframe cuts
  • every segment opening on its own
  • starting part way in
  • cancelling from another thread
  • 401 and 403 reported as refused, and unreachable sources
• mvp-core: 18 tests for the playlists, the codec names and the pacer.
• App: 21 hls_out tests that serve on loopback and fetch as a receiver would. They cover the master and media playlists, the segments, a held pacer, the sliding window, stopping, a refused login, video it can't copy, the subnet limit, and a source too short to cut. The app's tests also cover wifi, the commands and the argument table.
• Whole suites: the app (331), the plugin (175), mvp-core (577) and Vitest (2571) pass. Clippy on the workspace, lint and Prettier are clean.
• StrykerJS: 100% on the frontend lines this step changes on top of feat(airplay): send the playing item to an AirPlay receiver from iOS #323 (106 killed). Earlier runs had left four survivors, and each now has a test or is rewritten so nothing equivalent is left, except one disable comment on the "can't convert" default that says why.
• cargo-mutants: in place, on the changed lines of all four crates, with a re-run after each fix. Every viable mutant is caught; timeouts count as caught. The handle's Drop (frees memory, which no test can see) and two iOS-only app functions have skip comments.
• Pre-commit hook: skipped. Its Vitest run times out while three tracks build. I ran the suites by hand.

Needs the iPhone and the Samsung TV

On each one, check that the sound plays on the TV, not only AVPlayer's state.

1. M3U live raw TS, converted: does it start within a few seconds, and keep going for 30 minutes?
2. A DVB channel with MP2 audio, AC-3 VOD and DTS VOD, converted to AAC.
3. MKV H.264 VOD: resume at a position, seek, pause for a minute, then play.
4. The local network prompt when the converter's server starts.
5. Lock the phone for 10 minutes while converting.
6. Wi-Fi off, and no Wi-Fi at the start ("Connect to Wi-Fi to send this to the TV.").
7. Battery and heat over an hour of conversion.

• macOS
• Linux (Wayland)
• Linux (X11)

Review fixes (reviews/329.md)

Fixed in this PR:

• 2, 3 (load order): airplay/conversion.rs runs the converted loads. It's apart from iOS, so the tests and mutants reach it.
  • One load runs at a time. A later load or a stop makes an older one give up with superseded.
  • A new conversion opens its source only after the old one has closed its connection, waiting up to 5 s.
  • A conversion is kept only once the TV was told to play it. One that a stop overtook while it started is dropped.
  • The frontend runs one converted seek at a time, and only the last of those asked meanwhile. The player says when a seek on the TV fails.
• 4 (ended, reconnect): live HTTP opens with reconnect_streamed, reconnect_on_network_error and reconnect_on_http_error=5xx. A conversion that ends on its own with an error, or a live one that ends at all, shows as the item's error (refused or failed) through the plugin's new fail_converted.
• 5 (timestamps, keyframes):
  • Timestamps that jump back more than 1 s or forward more than 2 s are moved to carry on, and a packet whose DTS would go back is dropped.
  • A segment is cut at 8 s or 32 MB without a keyframe. A segment over 48 MB fails the conversion.
• 6 (start timeout): the converter exists before it opens, so a cancel stops an open that hangs. The start timeout cancels it, and the tests check that the source's connection closes. The start is ready at 3 segments or 6 s of media, so a long GOP still starts.
• 7 (VOD start): a movie's playlist has EXT-X-START:TIME-OFFSET=0,PRECISE=YES, and its pacer waits for the TV to play.
• 8 (live pause): a live pacer never stops. The lock screen and remotes can't pause a converted live channel, and Swift ignores them.
• 10 (audio changes):
  • The resampler is set up again when the audio's layout, rate or format changes.
  • The sample count follows the timestamps after a gap of more than 100 ms. A frame from before the count is dropped, so the AAC never goes back.
• 11 (confirm):
  • The convert question comes before the system picker, so a no leaves the sound here. If only the picked TV's check converts, it asks then.
  • Escape answers no, and so does the button going away while it asks.
• 12 (CODECS): H.264 whose profile can't be read is named avc1.640028 (High 4.0).
• 13 (window): the window keeps at most 15 segments, 50 s and 200 MB.
• 14, in part: remote seeks on a converted stream are ignored, since its timeline is a sliding window.
• 15.1: the target duration is a fixed 10 s.
• 15.2: a resume seeks from the source's first timestamp. MPEG-TS lands after its target, so the seek moves back until the first keyframe is at or before the start.
• 15.3: the position counts from the first segment's real start, which the C reports with each cut.
• 15.4: Converted sits above load's docs.
• 15.5: only playing runs a movie's pacer.
• 15.6: checked. The Channel model has no per-channel or per-provider user agent, so the direct path has none to pass on.
• 15.9: the conversion ends when the TV connection is lost.
• 15.11: the privacy policy says what AirPlay sends the TV, in both modes.
• 15.12: docs/development.md lists the FFmpeg dev packages.
• 15.13: see "How it was tested".

Follow-ups, not in this PR:

• 9 (suspension, accept loop): the accept loop is in cast's lan_server.rs. Retrying after ECONNABORTED or EMFILE and rebinding is cast's to change, and I've passed it on. Restarting a converted movie after the app was suspended comes after that.
• 14 (Now Playing): the lock screen's position and length on a converted movie still come from AVPlayer, and that needs the offset passed to Swift.
• 15.7: other interfaces than en0, such as a Personal Hotspot or Ethernet.
• 15.8: checking the local network permission before the TV connects.
• 15.10: the master's bandwidth from the source's bitrate.

How it was tested (review fixes)

On Linux, with no display and no device. CI's iOS job is the first compile of remux.c against MPVKit, and of airplay/ios.rs, the Swift and the iOS-only Rust.

• hls-remux: 23 tests, against the system FFmpeg 7.1. test-media/test_media.c writes H.264 (libx264) and MP2 in MPEG-TS for them, with a chosen GOP, channel count and timestamp offset. The app never builds it. The tests cover:
  • copying H.264 and where each segment starts
  • a resume from a file that starts at 50 s
  • a 10 s GOP cut at 4 s
  • jumps forward by 1000 s and back from 90 000 s
  • stereo turning mono mid-stream
  • a cancel that stops a hanging open
• mvp-core: the window's limits by count, seconds and bytes, the fixed target, EXT-X-START, readiness by time, and a pacer that waits while paused.
• App:
  • 28 hls_out tests: an H.264 source served and read back, a resume at the keyframe before, a live clock that never pauses, a movie that waits for the TV, a start timeout that hangs up on the source, and the segment byte cap.
  • 13 airplay::conversion tests: keeping, stopping, a stop while starting, failed starts and plays, quick loads collapsing into the last, waiting for the old source to close, a source that never closes, and failures reported only for the conversion that plays.
• Plugin: fail and fail_converted.
• Frontend: coalesced seeks, superseded, a lost TV ending the conversion, the confirm order, Escape and unmount, and VideoPlayer's seek failure.
  • StrykerJS kills every mutant on the changed lines.
  • The React Compiler can't compile a try with a throw, a ?? or a finally, so the AirPlay hook and button catch on the promise.

Needs the iPhone and the Samsung TV

On each, check that the sound plays on the TV, not only AVPlayer's state.

1. A converted M3U live channel (raw TS): it starts within a few seconds, and keeps going for 30 minutes. Turn the router's internet off for 10 s: it carries on, or the panel says the stream stopped.
2. A DVB channel with MP2 audio, and AC-3 and DTS VOD, converted to AAC. A channel that switches between stereo and mono, or 5.1 and 2.0, at an ad break keeps its sound.
3. MKV H.264 VOD:
   • Resume at 1:02:00: the TV starts at about 1:02:00, and the progress bar agrees with the picture.
   • Seek with the arrow keys five times quickly: one reconversion runs, and the provider sees no second connection.
   • Pause for a minute, then play.
4. A movie in MPEG-TS that starts at a large timestamp, resumed part way: it starts at the right place.
5. Pause a converted live channel from the lock screen and the TV remote: it doesn't pause.
6. Send a stream that converts, then tap Not now: the sound stays on the iPhone, and the picker never opens.
7. Stop, or leave the player, while "Starting on the TV…" shows: nothing starts on the TV later, and the next stream isn't refused.
8. The local network prompt when the converter's server starts.
9. Lock the phone for 10 minutes while converting.
10. Wi-Fi off, and no Wi-Fi at the start ("Connect to Wi-Fi to send this to the TV.").
11. Turn the TV off while converting: the conversion ends.
12. Battery and heat over an hour of conversion.

Checklist

• The pull request targets dev
• pnpm lint, pnpm test and cargo clippy --workspace --all-targets -- -D warnings pass
• User-facing changes are added to the Unreleased section of CHANGELOG.md
• No playlist URLs, logins or API keys in the code, tests or screenshots
• I have read the Contributor License Agreement and agree to it

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.