Add AGENTS.md: verified Tesla T4 (16GB) inference notes

[moduvoice]

What

Adds AGENTS.md — inference notes verified end-to-end on a real Tesla T4 (16GB, Turing,
sm_75). No code or behavior changes; this is a new documentation file only.

Why

The docs (docs/en/install.md, docs/en/inference.md) recommend "at least 24GB" but don't say
what actually happens on a 16GB card, or why. We ran the documented CLI (fish_speech/models/text2semantic/inference.py)
against the fishaudio/s2-pro checkpoint on a real T4 and found:

1. It OOMs on 16GB before generating a single token — even for a one-sentence input with no
   reference audio. Reproduced twice, identical error both times.
2. Root cause, confirmed by reading the code (not guessed): model.setup_caches(..., max_seq_len=model.config.max_seq_len, ...) is called unconditionally right after model load,
   and model.config.max_seq_len comes from the checkpoint's config.json (32768 for
   S2-Pro) — independent of your actual prompt length or --max-new-tokens. This
   pre-allocates a ~4.5GB KV cache (36 layers × 8 heads × 32768 × 128 × 2 bytes × 2 for K+V, bf16)
   before any text is even tokenized, on top of the ~8.5GB of bf16 weights.
3. A config-only workaround exists: lowering text_config.max_seq_len in the downloaded
   checkpoint's config.json (e.g. to 4096, still far more than any single-utterance TTS
   request needs) avoids the OOM entirely and runs comfortably in ~9.9GB. We verified this twice
   — same output, same timing both times.
4. Traced the attention-backend code path since Turing GPUs don't support flash-attention
   (sm80+ only): Attention.forward() has a branch that would force sdpa_kernel(SDPBackend.FLASH_ATTENTION)
   when mask is None, which would crash on T4 — but tracing the actual call path shows the
   inference loop's forward_generate() always builds an explicit mask, so that branch is dead
   code here, not a live risk. Actual generation is deliberately forced through SDPBackend.MATH
   by decode_n_tokens() — already T4-safe by the repo's own design.
5. Tested --half (fp16) explicitly: it does not crash (confirmed no NaN/Inf, unlike some
   other bf16-trained TTS models we've tested where forcing fp16 breaks sampling), but it also
   doesn't help the OOM or speed at all — fp16 and the default bf16 are both 2 bytes/param (same
   VRAM), and the decode loop is memory-bandwidth-bound rather than compute-bound (confirmed via
   the Bandwidth achieved log line, ~27.5-27.7 GB/s either way), so Turing's missing bf16
   Tensor Core path never becomes the bottleneck. --half is safe but not a useful lever here.
6. The int8/int4 quantization handlers in tools/llama/quantize.py are only reachable if the
   checkpoint directory path contains the literal substring int8/int4 (no CLI flag), target
   the older single-file checkpoint format rather than S2-Pro's sharded safetensors, and
   wouldn't touch the KV-cache allocation anyway — consistent with community reports in
   #1168 that even a third-party
   quantized checkpoint still used 21GB+.

None of this is a code bug — it's a legitimate design choice (simplicity of always allocating
for the declared max context) that just isn't documented anywhere, and the fix is a one-line
config edit rather than a code change, so we're contributing it as a note rather than a PR
against the inference code.

Verification (Tesla T4 16GB)

Environment: Tesla T4 16GB, driver 550.163.01, Python 3.10.12, torch 2.8.0+cu128, transformers 4.57.3.

Config	Result	Peak VRAM (nvidia-smi)	Gen time
max_seq_len=32768 (checkpoint default)	OOM, reproduced 2x (identical error)	15593 MiB both runs, right before crash	crashes before first token
max_seq_len=4096 (config edit only)	Success, reproduced 2x	9939 MiB both runs	23.75s / 23.67s (143 tokens both runs, ~6.0 tok/s)

Exact OOM error (reproduced twice, verbatim):

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 512.00 MiB. GPU 0 has a total capacity of 15.56 GiB of which 342.62 MiB is free. Including non-PyTorch memory, this process has 15.22 GiB memory in use.

Full details, raw metrics, and the T4 acceleration checklist (dtype/attention backend/int8/CUDA
graphs, each measured not assumed) are in AGENTS.md.

Test plan

• Ran documented CLI command as-is on Tesla T4 16GB (fails with OOM, 2x reproduced)
• Ran with the config-only max_seq_len workaround (succeeds, 2x reproduced, deterministic output both times due to --seed 42 default)
• Read llama.py/inference.py to confirm root cause and attention-backend behavior rather than assume
• Checked existing issues for duplicates (#1168 covers the general "doesn't fit in 16GB" report; this AGENTS.md adds the specific root cause and a working workaround, so it complements rather than duplicates that issue)
• No code changes — new doc file only