PROGRESS.md

Pipeline progress

Last updated: 2026-07-13

Target

Build and deploy an AWS-backed, durable batch pipeline for long audio files with music-first then voice separation, sound gates, persisted stems/metadata, single-consumer model queues, operational visibility, and fast manual review. Recursive separation of individual SFX is excluded from this milestone.

Datasets are persistent and accept any number of successive upload jobs; 100 was an example batch size, not a workflow limit.

Milestones

• Validate AWS credentials and the H100 host connection.
• Audit the existing SAM Audio API, timings, adaptive candidates, and three-bucket verification output.
• Record the production flow, data ownership, queue semantics, and review contract in ARCHITECTURE.md.
• Make the model cascade prompt-aware and deploy music-first defaults.
• Add the lightweight pipeline control-plane package and tests.
• Add CloudFormation for S3, DynamoDB, SQS/DLQs, and the EC2 role.
• Implement durable ingestion, 30s/5s chunking, and the sound gate.
• Implement the one-at-a-time SAM queue worker and stem persistence.
• Implement Audio Flamingo queue tasks for scene/music descriptions and voice transcription with diarization.
• Acquire and persist a reproducible AudioSet reference/calibration set.
• Implement job/queue dashboard and keyboard-first review UI.
• Add exact review backlog counts, dataset size/processing metrics, and the original-plus-stems source explorer.
• Add frequency-aware stereo/loudness reconstruction while preserving every raw mono stem, plus a dashboard playback toggle and smoothed trajectory plots.
• Join stored stereo stems, persist phase/level-sensitive waveform fidelity metrics, and graph the dataset-wide reconstruction score distribution.
• Persist and display each original's mono/stereo layout and encoding quality facts, with an idempotent historical backfill.
• Give pipeline, review, dataset, job, and source dashboard states durable URLs with refresh and browser-history restoration.
• Add a resumable general-YouTube (non-AudioSet) sampler with deterministic discovery, exact ten-second cuts, stereo/source-quality/silence gates, per-attempt provenance, hashes, and a full-dataset verifier.
• Require stereo input, gate SAM targets with source-scene presence, support single-stage inference, and use identity pass-through for pure SFX sources.
• Add persistent datasets, successive upload jobs, and reconciliation of uploaded/stale work from DynamoDB back into SQS.
• Provision AWS, deploy services, and run a multi-file end-to-end test.
• Verify recovery/reconciliation, review decisions, and all stored artifacts.
• Commit and push the verified implementation to qforge/main.

Current evidence

• AWS caller: account 088543363904, IAM user michal, region us-east-1.
• GPU host: i-0aed4af178083ce58, p5.4xlarge, H100 80 GB, reachable by SSH.
• Model services: sam-audio-small-tv in TF32/fp32 and Audio Flamingo Next in BF16 are both active on the H100. Their checkpoints live under /home/ubuntu/models and are not part of an image.
• SAM candidates expand 4 -> 8 -> 12. The default route is music-first; a failed cascade gets one voice-first retry and both attempts are persisted.
• AWS stack sam-audio-pipeline is live with a private/versioned/encrypted S3 bucket, point-in-time-recoverable DynamoDB table, three SQS queues and DLQs, and a least-privilege instance profile attached to the H100 host.
• AudioSet calibration set references/audioset/v1/ contains 8 private WAVs, manifest.json, and manifest.sha256.
• Seeded AudioSet validation set references/audioset/random-100-v1/ contains 100 private WAVs from 100 unique YouTube videos plus the manifest/checksum. All 100 records retain their official AudioSet start/end timestamps and labels; 28 unavailable candidates remain in the manifest as provenance. The successful clips total 996.795 seconds, and the 102 reference objects total 191,551,793 bytes.
• Dataset 991945c1f082446a9ff66d482838f964, job 3f0233f2442f4044a0f2967c68cb5a89, completed all 100 sources and chunks with zero failed chunks. It initially stored 284 audible stems (94 music, 91 voice, 99 SFX) and omitted 16 below-gate outputs (6 music, 9 voice, 1 SFX). The selected route was music-first for 63 sounds and voice-first for 37.
• After the presence repair, the completed random dataset contains 282 stems and 540,037,650 bytes of mapped stereo companions. All 282 companions were force-migrated to stereo v2 (or the identity path) without replacing their raw keys. Automatic verification places 159 stems in success, 54 in uncertain, and 69 in failure, leaving 123 review items.
• Browser verification confirms the review screen shows exact remaining, failure, and uncertain counts; the dataset screen shows 100 sounds and its storage breakdown; gated stems are absent from the stacked track view; the repaired writing source shows sfx only plus identity · no EQ; and the mono canary shows the explicit stereo-input skip reason.
• Live reference job ae31001eec754ee6850e3727134de84e processed the supplied 30-second chunk_ours/og.wav through the normal production handler and created raw plus stereo music, voice, and SFX objects. The rendered dashboard defaults to Stereo mapped, swaps only stem URLs when Raw is selected, and displays three 32-band/EMA-0.03 trajectory plots.
• The mapper was separately checked against the previously supplied raw humanvoices.wav: its smoothed pan moves from -0.33 (left) to +0.39 (right), matching the attached chunk-zero analysis. All three 30-second stems map in about 0.58 seconds locally; the fresh remote job mapped its three stems in about 1.02 seconds.
• Source 099_writing_mHf7COLGcD0_170.wav exposed both failure modes: its scene annotation correctly said no music and no voices, but the old cascade still created three success stems, and stereo v1 reduced music high-frequency energy from 8.7% to 0.8%. Stereo v2 retains 7.8% on the same raw stem. The live source was repaired from its stored scene evidence and now has one SFX-only stem; raw, mapped, and original normalized PCM are sample-identical. The dataset now indexes 282 stems (93 music, 90 voice, 99 SFX).
• Joined-stereo reconstruction is backfilled for all 100 stereo sources in seed 20260713 and all 98 stereo sources in seed 20260714. Their mean waveform similarity is 82.79% (7.11–100) and 88.02% (24.92–100), respectively. The repaired writing/SFX-only record scores exactly 100 with correlation 1.0 and 0 dB level error. Four additional reconstructable legacy sources (five chunks) were also migrated, for 202 full-source joins from 203 chunk joins; mono and sound-gated sources correctly have no joined artifact.
• Original audio profiles were backfilled for all 207 durable source objects across nine jobs. The only two records without profiles are upload records from an already-failed job whose source objects never reached S3. The current AudioSet set reports 98 stereo and two mono originals with exact codec, sample-rate, bit-depth, and bitrate facts.
• Local general-YouTube set youtube-random-1000-20260714 contains exactly 1,000 ten-second WAVs from 1,000 unique direct YouTube videos selected across 288 randomized mix-biased queries. Its independent audit passes with no failures: every output is stereo PCM16/48 kHz, all source streams are stereo at 44.1 kHz or better and at least 120.084 kbps, and every loudness, silence, true-stereo, clipping, duration, uniqueness, existence, and SHA-256 gate passes. The audio payload is 1,920,078,000 bytes; no AudioSet record metadata or AudioSet candidate source was used.
• M2D fragment validation scored all 1,000 local general-YouTube clips with the official AudioSet-fine-tuned checkpoint. Policy spoken_dialogue_instrumental_background_m2d_v2 accepts 429 clips with temporally overlapping spoken dialogue and active background, while rejecting 343 for singing, rapping, choir, or other vocal music. The listening output preserves all 429 accepted clips and includes a 272-clip balanced subset with 136 music-led and 136 effects/ambience-or-mixed clips.
• Live mono-filter job 5ec26b1bb28b420a9245e21d475d26be completed with non_stereo_input, one input channel, zero chunks, zero stems, and zero model tasks. A live targets=voice API canary produced only voice and residual artifacts in one 12.8-second SAM stage.
• Multi-file job bd85486f9d314c258956cac9b81a730f completed: one 30-second audible source produced exactly one chunk and three stems; one silent source was sound-gated without GPU work; all three Audio Flamingo tasks completed.
• That job triggered both cascade orders. Music-first scored 18.03 versus 8.11 for voice-first, so the policy retained music-first and stored both timing/score summaries.
• Recovery job 1d07979ee96a45a98cf220b102fd7006 was uploaded without the completion callback. Reconciliation found the S3 object, enqueued ingestion, and completed it as sound-gated. All three queues were empty afterward.
• Local and remote validation: 39 pipeline tests pass; the existing 12 SAM batching tests also pass.
• Docker build execution remains unverified because neither the local Mac nor the current H100 host has a Docker daemon. Both Dockerfiles remain model-free.

Decisions and open deployment work

• DynamoDB is the job/stem/review source of truth; SQS is transport, not state.
• S3 stores all audio and metadata; the model image continues to exclude model weights.
• The control plane is a separate lightweight package/image from the SAM model server.
• The H100 instance uses the stack-managed least-privilege IAM instance profile; no long-lived AWS key is stored in the service environment.
• The initial panel remains private behind an SSH tunnel until authentication is explicitly configured.