YaRN rope scaling, 4-bit F4 packed KV cache, and hybrid paged attention for Qwen3.5-Next long-context serving

[JLFN]

Summary

This PR merges the Qwen3.5-Next long-context serving work back to master. It adds YaRN rope scaling pinned to the llama.cpp reference implementation, a 4-bit F4 packed KV cache for paged attention, hybrid paged-KV allocation that restricts the paged cache to the full-attention layers of the hybrid model, host-memory parking of recurrent prefix snapshots, a halved paged-prefill chunk size for the 262144-token window, a MoE router topk fix on CUDA, and the lockfile security remediation. The target workload is serving the Qwen3.6-14B-A3B Q4_K_M GGUF (qwen35moe, 40 layers, 90 experts, full_attention_interval 4) with up to 320000 tokens of context on a 12 GB consumer GPU.

Origin and included prior work (feat/qwen35-native-context-fix)

Items 2 through 7 below - the F4 cache, the hybrid paged-KV allocation, the recurrent-snapshot host parking, the paged-prefill chunk change, the MoE router topk fix, and the lockfile remediation, with their tests - were developed and verified earlier on the fork branch feat/qwen35-native-context-fix (tip f93428e, pushed to JLFN/mistral.rs). That branch has no pull request of its own: its 34-file, +1623/-161 diff against master is a strict subset of this PR's 51-file diff, byte-identical in every file except the YaRN additions on top (items 1 and 8 below) and a small extension to f4_cache_storage.rs. The YaRN long-context work (item 1) is the new layer this PR adds over that native-context base, and both together are squashed into the single commit b7f9c90.

Changes by area

1. YaRN rope scaling for the Qwen3.5 partial rotary (new)

• layers.rs: RotaryEmbedding::new_partial_yarn resolves the rope the same way llama.cpp does: get_mscale cancellation in llama-context.cpp against the 1 + 0.1*ln(scale) re-fold inside the ggml rope_yarn kernel, netting a table scale of 1.0199427025 for factor 1.220703125, with corr_dims low/high ramp from beta_fast 32 / beta_slow 1 exactly as ggml.c computes it.
• models/qwen3_next.rs: Config gains a rope_scaling field (rope_type, factor, original_max_position_embeddings, beta_fast/slow, mscale, mscale_all_dim); the model builds new_partial_yarn when the config carries a yarn entry and bails on unsupported rope types.
• New RopeOverride type (rope_override.rs), threaded from the CLI through ModelSelected, the GGUF loader, and TOML selector into config synthesis.
• gguf/normal_config.rs: synthesizing a config under a forced YaRN override neutralizes any embedded GGUF rope.scaling metadata (which would otherwise make the registered builder reject the checkpoint), then inserts the yarn rope_scaling block and sets max_position_embeddings to the resolved target context.
• CLI: --rope-scaling yarn|none, --rope-scale, --yarn-orig-ctx, --override-ctx on the GGUF model format options of run and serve.
• Golden tests (yarn_rotary.rs) probe positions 262144, 262145 and 319999 against the exact llama.cpp formulas, matching to <= 5e-12.

2. 4-bit F4 KV cache (paged-attn + core)

• candle-core cannot allocate or cast DType::F4 on CUDA in the pinned rev (verified against source; see F4-KV-STORAGE-ASSESSMENT.md), so the cache is stored as packed U8 tensors with custom kernels.
• PagedCacheType::F4 added; cache_engine block-shape math uses 32 values per 16-byte cell row for K and head_dim/2 byte-rows for V.
• reshape_and_cache_f4_kernel writes symmetric 4-bit values (nibble - 8) * scale: one scale per 32-value K cell and one scale per token per V head, with race-free 2-pair-per-thread byte writes.
• The v1/v2 paged_attention kernels dequantize F4 cells inline during attention; gather_kv_cache and copy_blocks gain U8/F4 paths; kv scales become per-cell tensors guarded by a Mutex.
• Storage cost: 5 KB/token for the 10 paged (full-attention) layers vs 20 KB/token for BF16, a 4x reduction; 5.5 KB/token measured at serve.
• Tests: f4_cache_storage.rs (candle F4 limits, pack/unpack tolerance), f4_kv_layout.rs (CPU mirror of the kernel byte indexing), f4_gpu.rs (CUDA kernel round-trip vs CPU reference).

3. Hybrid paged-KV allocation for Qwen3.5-Next

• qwen3_next and the Qwen3.5 vision text model expose model_config via HybridPagedKvCacheConfig; paged KV blocks are allocated only for the 10 full-attention layers of the 40-layer hybrid instead of all 40.
• Test hybrid_layer_types_mask_only_the_10_full_attention_layers pins the mask to layers 3, 7, 11, 15, 19, 23, 27, 31, 35, 39.

4. Recurrent prefix snapshots parked in host memory

• prefix_cacher.rs moves conv_state and recurrent_state of paged recurrent prefix snapshots to Device::Cpu before caching. Each snapshot holds every recurrent layer's F32 state (~63 MB for Qwen3-Next's 30 GDN layers), so GPU residency would accumulate ~1 GB of VRAM per cache fill.

5. Paged prefill chunk halved 4096 to 2048

• pipeline/mod.rs: DEFAULT_PAGED_PREFILL_CHUNK_SIZE is now 2048, sized for the 262144-token window.

6. MoE router CUDA topk fix

• ops.rs: cuda_topk operates on a contiguous copy of the input instead of a final-logits row view, restoring full-row topk for MoE routing on CUDA.

7. Dependency security remediation

• Cargo.lock-only updates (no manifest changes): anyhow, crossbeam-epoch, memmap2, quinn-proto (CVE-2026-25800), rand, tar.
• The remaining 7 vulnerable packages (10 advisories) are recorded as accepted risk in OSV-EVIDENCE.md with per-package reasoning: TLS-path issues in aws-lc-sys not reachable on the plain-HTTP deployment, pyo3 not in the CLI deliverable, and unmaintained build-time/transitive crates with no fix version.

8. Docs

• YARN-SOURCE-VERIFICATION.md: llama.cpp source-line evidence for the rope pipeline (arg parsing, yarn_attn_factor cancellation, ggml rope_yarn re-fold, IMROPE-to-NEOX equivalence, corr_dims) plus the value-level comparison and the Phase B rebuild / Phase C serve evidence.
• F4-KV-STORAGE-ASSESSMENT.md: candle-core storage limits, the chosen packed format, and authoritative per-token cache numbers.
• OSV-EVIDENCE.md: remediation list and accepted-risk set.

Verification

• cargo test -p mistralrs-core --test yarn_rotary: 4 passed (positions 262144, 262145, 319999 match the llama.cpp pipeline to <= 5e-12; a Python reference reproduces llama.cpp to machine precision).
• cargo test -p mistralrs-core --lib qwen35moe_yarn_override: 2 passed (synthesis of the 320000/262144 override including the wins-over-embedded-metadata case).
• F4 tests: f4_cache_storage (CPU + CUDA storage limits), f4_kv_layout (CPU byte-layout mirror), f4_gpu (CUDA kernel vs CPU reference round-trip).
• Binary rebuilt with cuda + flash-attn; serve boot shows the F4 packed u8 cache, the yarn flags honored, and 320000 available context.
• Serve evidence on RTX 4070 SUPER 12 GB with Qwen3.6-14B-A3B Q4_K_M FableVibes: run 1 reached prompt ~252k then CUDA_ERROR_OUT_OF_MEMORY at GPU peak 11807/12282 MiB; run 2, after freeing ~0.5 GB of non-server GPU consumers, reached prompt 275837 (total ~280k tokens) with a 99.83% prefix-cache hit rate, stable F4 cache and no overflow, before being stopped. The full 320000 completion is not yet verified.
• OSV scan at PR time: 800 packages scanned, 7 vulnerable (10 advisories), all the documented accepted-risk set, no new findings.

Known limitations

• The F4 cache exists only in the Standard paged-attention layout/kernels: the engine forces the Standard backend when the cache type is F4 (FlashInfer is disabled), and any CPU-hosted layer in the device map disables paged attention entirely by design, so F4 and host-RAM offloaded KV are mutually exclusive.
• The F4 startup log over-reports the reserved cache budget 2x (blocks are stored packed U8, 16 bytes per 32 values); the correction is cosmetic and frees no real memory.
• F4-vs-BF16 logit equivalence at long context has not been measured; the format's per-value error bound is max_abs/16.
• 320000-token full-window serving on this 12 GB card is bounded by VRAM: the F4 cache ceiling observed is ~275k-280k tokens; the CPU/RAM layer-offload route that would exceed it is a planned follow-up outside this PR.

Commits

One squashed commit (b7f9c90) ahead of master: the full substantive long-context work condensed into a single conventional commit, 51 files changed, 2182 insertions, 192 deletions. The session-handoff document and the separate CPU-offload experiment branch are intentionally not part of this pull request; the offload work is a planned follow-up.

[github-actions]

Code Metrics Report

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Language              Files        Lines         Code     Comments       Blanks
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 C Header                 24         4497         3151          790          556
 CSS                       3          281          252            5           24
 CUDA                    124        26446        21759         1707         2980
 Dockerfile                1           35           19            9            7
 HTML                      2           27           27            0            0
 JavaScript                3          582          567           12            3
 Jinja2                    7          694          656            5           33
 JSON                     27        18278        18275            0            3
 Makefile                  1           18           16            0            2
 MDX                      37         6771            0         4994         1777
 Metal Shading Lan|       37        14416        11408         1136         1872
 PowerShell                1          657          571           31           55
 Python                  152        12617        10486          480         1651
 Shell                     3         1071          852          115          104
 Plain Text               53        10687            0         9209         1478
 TOML                     28         1397         1213           43          141
 TypeScript               11         1658         1418           66          174
 YAML                      3           25           23            2            0
─────────────────────────────────────────────────────────────────────────────────
 Jupyter Notebooks         3          122           83           23           16
 |- Markdown               1           60           30           22            8
 |- Python                 1          122          113            1            8
 (Total)                              304          226           46           32
─────────────────────────────────────────────────────────────────────────────────
 Markdown                275        12394            0         9265         3129
 |- BASH                  25          306          225           48           33
 |- Dockerfile             2           14           12            0            2
 |- JSON                   6          289          289            0            0
 |- PowerShell             1            1            1            0            0
 |- Python               135         7349         6119          306          924
 |- Rust                  62         3845         2851          396          598
 |- TOML                   7          116           97            0           19
 (Total)                            24314         9594        10015         4705
─────────────────────────────────────────────────────────────────────────────────
 Rust                    728       399029       360689         5489        32851
 |- Markdown             427        10136          452         8486         1198
 (Total)                           409165       361141        13975        34049
─────────────────────────────────────────────────────────────────────────────────
 Svelte                   19         1974         1832           50           92
 |- CSS                    1            4            4            0            0
 |- JavaScript            19          921          767           25          129
 (Total)                             2899         2603           75          221
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Total                  1542       536839       444257        42715        49867
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[JLFN]

Superseded by #2376 (fresh pull request from the same branch feat/qwen35-long-context with the same single commit b7f9c90 and description).