fix(qwen3.5): better support for hybrid models and auto device map - #2373
Merged
Conversation
Snapshots were keyed on exact token multiples of the block size, which prefill never lands on since it advances by the whole prompt in one step. The first key written was always past the largest reachable block offer, so no lookup could ever hit and every decode-time snapshot leaked. Block-align the prefill chunk tail so the existing recurrent_boundaries path fires at the offer size, and restrict the engine hook to prompt steps.
Device mapping derives a budget that subtracts the activation reserve, then the pipelines called calculate_cache_config again with Utilization and no model weight size, recomputing 0.9 * total - used. Since used already covers the weights, the reserve was dropped and the leftover was always a flat fraction of the card regardless of model or workload. Write the derived budget back so both consumers read the same number.
Sizing multiplied a single per-layer element count by the number of paged layers, so any model whose layers differ had to flatten itself to an average first. Gemma4 summed per layer then divided by the layer count purely to fit that interface. Sum per layer instead, with layers that hold no paged cache reporting None.
The activation reserve is predicted from max_seq_len and never checked against what a prefill costs. Run one synthetic prefill after the cache is allocated and log the resident bytes next to the prediction. Reported only; nothing is sized off it yet. The mempool release threshold keeps freed memory out of the driver, so free memory after the forward is already the high-water mark.
A chunk late in a long sequence attends over the whole KV cache, so measuring only at offset zero would miss any growth. Probe again further in and report the slope. Measured flat on both a dense and a hybrid model up to 64k of context, which is what paged attention should give: the KV is streamed, not materialised.
The repacked weights are a second full copy held for the process lifetime. Leaving it on by default spent that memory on every run, and because the repack claimed it after the KV cache was already sized, it silently switched itself off whenever the card was full. Same binary, same model, different behaviour depending on what else was resident. Default it off behind MISTRALRS_GGUF_AFFINE_BACKEND=on, and when it is on subtract the exact size as a KV cache budget term rather than correcting the cache afterwards.
OpenAI sends tool_calls[].function.arguments as a JSON string, while chat templates are written against the map transformers passes. Templates doing `arguments|items` therefore failed to render with "cannot convert value into pairs", which surfaced to clients as a 200 carrying the Jinja error. Parsing was already done, but only for three templates matched by name. Key it on the template actually iterating arguments instead.
Code Metrics Report━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Language Files Lines Code Comments Blanks ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ C Header 24 4497 3151 790 556 CSS 3 281 252 5 24 CUDA 124 26446 21759 1707 2980 Dockerfile 1 35 19 9 7 HTML 2 27 27 0 0 JavaScript 3 582 567 12 3 Jinja2 7 694 656 5 33 JSON 27 18278 18275 0 3 Makefile 1 18 16 0 2 MDX 37 6771 0 4994 1777 Metal Shading Lan| 37 14416 11408 1136 1872 PowerShell 1 657 571 31 55 Python 152 12617 10486 480 1651 Shell 3 1071 852 115 104 Plain Text 53 10687 0 9209 1478 TOML 28 1396 1212 43 141 TypeScript 11 1658 1418 66 174 YAML 3 25 23 2 0 ───────────────────────────────────────────────────────────────────────────────── Jupyter Notebooks 3 122 83 23 16 |- Markdown 1 60 30 22 8 |- Python 1 122 113 1 8 (Total) 304 226 46 32 ───────────────────────────────────────────────────────────────────────────────── Markdown 275 12389 0 9260 3129 |- BASH 25 306 225 48 33 |- Dockerfile 2 14 12 0 2 |- JSON 6 289 289 0 0 |- PowerShell 1 1 1 0 0 |- Python 135 7349 6119 306 924 |- Rust 62 3845 2851 396 598 |- TOML 7 116 97 0 19 (Total) 24309 9594 10010 4705 ───────────────────────────────────────────────────────────────────────────────── Rust 728 398957 360640 5474 32843 |- Markdown 425 10117 452 8470 1195 (Total) 409074 361092 13944 34038 ───────────────────────────────────────────────────────────────────────────────── Svelte 19 1974 1832 50 92 |- CSS 1 4 4 0 0 |- JavaScript 19 921 767 25 129 (Total) 2899 2603 75 221 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Total 1542 536742 444207 42679 49856 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ |
Defaulting the backend off left the packed kernels untested: the unit tests drive that path directly and silently fell through to canonical dispatch. Default to on under cfg(test); the env var still overrides either way.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.