[Pfannkuchensack]

Summary

SD 3 / 3.5 LoRAs could not be used, and most installed as another model:

• diffusers and PEFT files went to the Qwen-Image probe, which shares the attention names.
• kohya files that also train both text encoders became SDXL.
• DreamBooth repos were "unknown".
• SD 3 had no LoRA loader and its denoise node did not patch.

After this PR, SD 3 LoRAs are identified with their size and applied:

• Formats:

  • diffusers, ai-toolkit and PEFT (transformer., base_model.model.), including attn2, the conv pos_embed.proj and top-level modules.
  • SimpleTuner LyCORIS (lycoris_ with flattened names, LoKR).
  • kohya sd-scripts with the original MMDiT names. The fused qkv is split into thirds, and the two adaLN projections stored with swapped halves are unswapped. Both match diffusers' single-file converter, which a drift test checks.
  • DreamBooth folders, which read the top-level weights, not checkpoint-* copies.
  • Text encoders: CLIP-L and CLIP-G parts apply to their encoders. T5 parts are dropped with a warning.
  • Layouts that cannot be applied are refused at install: DoRA, unknown layers, incomplete layers, LoKR/LoHA on layers that must be split or swapped, layers at both SD 3 widths, blocks beyond the size's depth (24 Medium, 38 Large), and files that name only SD 3 modules but carry no layer that gives the width.

• Recognition: SD 3 is identified by its widths (1536 Medium, 2432 Large), because Qwen-Image uses the same attention names. Attention, MLP-input and modulation layers all give the width. Kohya-style files that keep diffusers' names (lora_unet_transformer_blocks_0_attn_to_q…) are read too. The Qwen-Image probe now rejects SD 3 widths, module names Qwen-Image never has (ff.net, norm1.linear), incomplete attention layers, and CogView4's attn1. The generic SD/SDXL probe rejects kohya joint_blocks.

• Size: new optional variant (sd3_medium / sd3_large) on SD 3 Diffusers mains and SD 3 LoRAs, set at install. SD 3 Medium and 3.5 Medium share sd3_medium. The webv2 concept picker hides LoRAs of the other size when both sides record one.

• Nodes: new sd3_lora_loader and sd3_lora_collection_loader with transformer, CLIP-L and CLIP-G fields. sd3_text_encoder patches CLIP-L and CLIP-G with separate prefixes. sd3_denoise (1.2.0 → 1.3.0) refuses, before the transformer moves to the GPU (it is already in RAM, and text encoding can have run):

  • a LoRA of the other size, by recorded variant or by shape;
  • a LoRA with dual attention on SD 3 Medium;
  • any layer, LoRA, LoKR, LoHA or full, whose update does not fit the module it patches. LoKR and LoHA shapes are read from their factors; no delta is built for the check.

  It patches in place, or through a sidecar with FP8 Storage, and then reserves the patches.

• webv2 and docs: SD 3 joins LORA_SUPPORTED_BASES, the graph routes transformer and CLIP-L/G through the collection loader, and the size has labels and options in the model manager. The docs add a LoRA section to the SD 3 page and update the format matrix and the Generate panel page.

Related Issues / Discussions

Closes #9741

QA Instructions

Automated:

• CUDA_VISIBLE_DEVICES="" uv run --no-sync pytest -n 4 tests/backend/model_manager tests/backend/patches tests/backend/architectures tests/model_identification tests/app: 10,903 passed. The only failures were the bitsandbytes nf4/int8 tests, which need CUDA; with the GPU visible, tests/backend/model_manager/load/model_cache/torch_module_autocast passes. Each new test file also passes on its own.
• New coverage:
  • The kohya conversion matches convert_sd3_transformer_checkpoint_to_diffusers on random MMDiT weights, for Medium with attn2 and for Large.
  • The real key layouts land on the production-size transformer.
  • Probe exclusivity: each SD 3 fixture is accepted by exactly one config. The Qwen, CogView4, FLUX (incl. OneTrainer flat names), Anima and other fixtures are not taken as SD 3.
  • Stripped real files are in the identification suite: kohya Clybius/mspaint-style-sd3.5medium and a DreamBooth folder.
  • Records without a variant stay readable.
  • Denoise: the fit refusal comes before the transformer moves to the GPU, for every layer type, and the FP8 Storage sidecar both computes the LoRA and reserves it.
  • invoke() hands CLIP-L and CLIP-G their own layers.
• ruff check/format, pnpm -C invokeai/frontend/api lint and test, pnpm -C invokeai/frontend/webv2 lint, and the webv2 generation/model tests (incl. settings/taxonomy and the concepts browser test): passed.
• Locally, the HF download tests fail only when HF_ENDPOINT points at a mirror that lacks the Xet endpoints; they pass without it.

Manual E2E on an RTX 4090 with SD 3.5 Medium (Diffusers), same seed with and without LoRA:

• LoRAs:

  • AdamLucek/sd3.5M-jareklfoto-LoRA-DreamBooth (diffusers)
  • bghira/sd35m-sfwbooru-lycoris (SimpleTuner LoKR)
  • jieliu/SD3.5M-FlowGRPO-PickScore (PEFT)
  • Clybius/mspaint-style-sd3.5medium (kohya, with attn2 and the swapped adaLN)
  • linoyts/Yarn_art_SD3_LoRA (diffusers with CLIP-L/G)
  • tensorart/stable-diffusion-3.5-medium-turbo (4 steps)

  Each applies visibly. Yarn with the text-encoder part differs from yarn with the transformer part only. The turbo LoRA gives a clean image at 4 steps where the base model does not.

• FP8 Storage: mspaint and yarn with CLIP.

• Logs: the server log contains no Failed to find module for LoRA layer key or Skipping LoRA layer.

• Browser (webv2): for SD 3.5 Medium, the concepts slot shows only Medium LoRAs, and a LoRA marked sd3_large is hidden. Sending a Large LoRA through a hand-built graph fails before loading with "LoRA '…' was trained for SD 3.5 Large and does not fit this SD 3 / 3.5 Medium model."

Review

The review findings are resolved:

• The speculative lora_transformer_ prefix is gone. No SD 3 file uses it, and it tied with FLUX OneTrainer files.
• Incomplete LyCORIS layers are refused at install instead of failing at conversion.
• Whether a kohya adaLN is swapped is decided by name and shape, so a LoKR on an unswapped modulation converts.
• The size refusal is readable and happens before the transformer moves to the GPU.
• A mismatched LoKR or LoHA is refused with the same message instead of failing at the first forward.
• Some SD 3 layouts that still installed as Qwen-Image are now SD 3 or refused with a reason.
• Re-identify names the reason when a probe refuses the file, and leaves the record as it was (shared change in ModelManagerService.reidentify_model).
• The probe tests list the exact accepted configs again.
• override_fields is no longer mutated.
• One width-to-size helper serves the probe, the main config and the denoise.

Remaining limitations:

• OneTrainer SD 3 files that use lora_transformer_ and also train both text encoders are still taken as SDXL, as before this PR.
• A LoKR under FP8 Storage is applied in weight space and is slower than a plain LoRA.
• SD 3.5 Large is covered by meta/ground-truth tests only; no Large checkpoint was run.
• Separate default settings per SD 3 size are not part of this PR.
• The sidecar reservation stays calc_size() plus 1 GiB: main's lora_sidecar_bytes reads in_features/out_features, which the Conv2d pos_embed.proj lacks.
• The fit check does not check convolution factorizations (kernel dimensions or a Tucker core), which only pos_embed.proj has.
• A diffusers-layout CogView4 LoRA identifies as Anima; this predates the PR.

Compatibility / Rollout

• Persisted configs: variant is optional and defaults to None, so stored SD 3 records stay valid and discriminator tags do not change. New installs record it. Existing mains and LoRAs get it, and misfiled LoRAs get the right base, through Re-identify in the Model Manager. A file the probes now refuse keeps its record, and Re-identify reports why. Re-identify also resets a model's default settings, which the docs note. The picker and the denoise accept records without a size, and the denoise then checks the shapes.
• API: SD3VariantType joins the variant unions; the OpenAPI schema and types are regenerated.
• Merging with the ERNIE-Image ([enhancement]: LoRA support for ERNIE-Image #9739) and Ideogram 4 ([enhancement]: LoRA support for Ideogram 4 #9740) LoRA PRs: whichever merges later needs generateGraphNodeTypes.json regenerated.

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Meaningful regression coverage added / updated where needed; obsolete tests/code removed
• Persisted-state and API changes include required migrations / compatibility validation
• Relevant performance/efficiency opportunities considered; material claims have evidence
• Material review findings resolved and relevant checks rerun
• Documentation added / updated (if applicable)
• Updated What's New copy (if doing a release after this PR)

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

[lstein]

Reviewed at a3dcf5e43c. I found no wrong images, leaks or conversion errors, but the items under "Blocker" and "Should fix" need attention before merge.

Blocker

Merge conflicts with main in configs/lora.py, 03.model-format-support.mdx and schema.ts (after #10094). schema.ts needs regenerating. In a scratch merge, keeping both sides of lora.py passed the config, SD 3 and FLUX.3 tests, and the probes from #10061 and #10094 did not overlap with SD 3 on the keys tried.

Should fix

1. A mismatched LoKR crashes instead of being refused (sd3_denoise.py:427). The shape check in _raise_unless_loras_fit only covers LoRALayer.

   • Trigger: an SD 3 main installed before this PR has no recorded size, so a Large LoKR passes the check. An SD 3 Medium LoKR that trains norm1.linear on SD 3.5 Medium (9216×1536 vs 13824×1536) passes too.
   • Result: RuntimeError: shape '[1536, 1536]' is invalid for input of size 5914624, reproduced at the first forward through the autocast wrappers. Weights are restored, but sd-3.mdx:101 promises a refusal when generation starts.
   • Fix: compare each layer's delta shape, not just low-rank layers'.

2. Some SD 3 files still install as Qwen-Image (lora.py:930). The Qwen probe only rejects what is_sd3_lora recognises. All of these identify as LoRA_LyCORIS_QwenImage_Config at width 1536:

   • transformer.transformer_blocks.0.ff.net.0.proj.lora_A/B.weight
   • transformer.transformer_blocks.0.norm1.linear.lora_A/B.weight
   • a lone transformer.transformer_blocks.0.attn.to_q.lora_A.weight, which also skips the promised refusal of incomplete layers
   • lora_unet_transformer_blocks_0_attn_to_q.lora_down/up.weight, although the comment above _FLAT_TEMPLATE_RE says kohya-style trainers that keep diffusers names are covered

   From reading the code, these layouts already went to Qwen before this PR. I did not trace what generation does with such a record. Qwen-Image never names ff.net or norm1.linear, so the probe could reject them by name and also check the width of any 2-D layer.

3. Re-identify can't fix a misfiled LoRA that the new probe refuses (model_manager_default.py:88). When result.config is None, it raises "Unable to identify model format", which drops the refusal reason in invalid_matches, and the old Qwen record stays in place. sd-3.mdx tells users Re-identify fixes misfiled LoRAs. This is from reading the code, not from a run. Please surface the reason, and have the docs say that a refused file stays as it was.

4. a3dcf5e43c weakens a test (test_sd3_lora_probe_independence.py:160). Exact accepted-config lists became "these 3 configs are absent", though the old expectation still holds at this commit: [Anima] for diffusers and [] for kohya. As written, a new config claiming these files would pass. Please restore the exact lists. This also shows that a diffusers-layout CogView4 LoRA identifies as Anima, even though CogView4 has no LoRA support.

Minor

• Untested paths (settings.test.ts:688): the code handles variant: null correctly, but the test only uses undefined, and the API sends null. Changing the check to === undefined passes the generation, models/core and image-actions tests, and so does hiding Medium LoRAs from Large mains.
• Misleading message (sd3_denoise.py:432): a low-rank SD 3 Medium LoRA on SD 3.5 Medium is told "another SD 3 size", but both are Medium.
• Loose install checks: mixed 1536/2432 widths install as Medium because only the first attention layer is read, and transformer_blocks.99.attn.to_q installs too. The denoise check catches low-rank layers at generation.
• 1-D LoKR factors (sd3_lora_constants.py:163): with otherwise valid Qwen layers, a file with 1-D lokr_w1/lokr_w2 becomes Unknown_Config. _factor_shape assumes 2-D tensors, which is the likely cause. I know of no trainer that writes this.
• Refusal timing: sd-3.mdx:101 says "before anything is applied", but CLIP patching and encoding can run first. The PR text says "before it loads the transformer", but it is already loaded into RAM (sd3_denoise.py:218); only the move to VRAM is avoided.
• Sidecar reservation: calc_size() plus a flat 1 GiB undercounts F16 LoRAs cast to the inference dtype, though the margin should absorb that. main now has lora_sidecar_bytes for this, so it is worth checking whether it fits after merging.
• webv1: ModelVariantSelect has no sd3_* options, so it should render blank for sized SD 3 records (code read only).

What held up

• Conversion: the kohya path matched convert_sd3_transformer_checkpoint_to_diffusers on 51 layers with no differences, including the qkv split, the adaLN swap and alpha scaling. Forward passes matched weights plus delta for direct, sidecar and FP8 hook application, and weights were restored exactly.
• Runtime: the refusal path, patch restore, iterate reuse and the conditioning cache key showed no defects, and CFG does not double-patch.
• Records: rows without a variant load fine, and the variant survives a JSON round trip.
• Frontend and API: canvas graphs have no dangling or doubled edges, and a schema regenerated from this commit's Python equals the committed openapi.json.

Not run: real SD 3 checkpoints, GPU or ROCm, or browser tests. The scratch merge got backend tests only, with no contract regeneration. Alpha handling for kohya split_qkv merged files is unverified.

[Pfannkuchensack]

Thanks, all addressed (merge 10ec733, then 37ea8d3 … dde1ad0):

Blocker: main merged; configs/lora.py keeps both the SD 3 and FLUX.3 configs, the format matrix is fixed, and the contracts are regenerated.

Should fix

1. LoKR/LoHA fit: _raise_unless_loras_fit checks every layer type, not just LoRALayer. LoKR and LoHA shapes come from the factor shapes (layer_delta_shape), with no delta built for the check, so a fit check costs nothing even for a full-coverage LoHA in fp16. Your 9216×1536 vs 13824×1536 norm1.linear LoKR is refused with the documented message. Convolution factorizations on pos_embed.proj (kernel dimensions or a Tucker core) are not checked.
2. SD 3 files going to Qwen-Image: none of your four layouts are claimed by Qwen-Image any more:
   • SD 3 recognition reads kohya files that keep diffusers' names (lora_unet_transformer_blocks_0_attn_to_q…), and takes the width from attention, MLP-input and modulation layers.
   • The Qwen-Image probe rejects ff.net / ff_context / norm1(_context).linear, incomplete attention layers, and low-rank attention layers that are not 3072×3072.
   • A lone half-layer is refused with a reason.
   • The Qwen-Image fixtures (diffusers, kohya, ComfyUI, raw PEFT, LoKR) still identify as Qwen-Image.
3. Re-identify: when a probe refuses the file, the error now names the reason and the record stays as it was (ModelManagerService.reidentify_model, with a service test). sd-3.mdx says so. This also covers the ERNIE-Image case from feat(ernie-image): support LoRAs #9822.
4. Probe-independence test: the exact lists are restored ([Anima] for the diffusers CogView4 layout, [] for kohya), with a comment that the Anima match is a pre-existing misidentification.

Minor

• settings.test.ts has variant: null cases and a Large main case: Medium LoRAs are hidden and Large LoRAs are shown.
• When both sizes are Medium, the message says "different SD 3 checkpoint layout".
• Install refuses layers at two widths and blocks beyond the size's depth (24 Medium, 38 Large).
• 1-D lokr_w1/lokr_w2 factors are read as one row, as torch.kron does.
• The refusal timing in sd-3.mdx and the PR text is corrected: the refusal comes before the transformer moves to the GPU, and text encoding can have run.
• Sidecar reservation: I kept calc_size() + 1 GiB. lora_sidecar_bytes reads in_features/out_features, which the Conv2d pos_embed.proj lacks, so it would need a Conv2d guard in the shared helper first.