You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
enable_group_offload(offload_type="block_level") fails on LTX / LTX-2 / LTX-2.5 pipelines (vae.decode, audio_vae.decode, LTX2VocoderWithBWE, and nested block_modules) #15250
Enabling pipeline-level block group offloading (pipe.enable_group_offload(..., offload_type="block_level", num_blocks_per_group=1)) on LTX / LTX-2 / LTX-2.5 pipelines fails at decode time with CPU/GPU device mismatches (and ValueError when use_stream=True is combined with _group_offload_block_modules on models that also hold root-level buffers):
AutoencoderKLLTXVideo and AutoencoderKLLTX2Video do not define _group_offload_block_modules = ["encoder", "decoder"], so encoder and decoder fall into the root unmatched_group whose hook is only triggered on vae.forward(), whereas pipelines invoke vae.decode() / vae.encode() directly.
AutoencoderKLLTX2Audio not only lacks _group_offload_block_modules = ["encoder", "decoder"], its LTX2AudioEncoder / LTX2AudioDecoder stages are bare nn.Module() containers iterated via attribute access (stage.block, stage.upsample) rather than stage(...), bypassing GroupOffloadingHook registered on self.down / self.up entries.
LTX2VocoderWithBWE wraps vocoder and bwe_generator (LTX2Vocoder instances) without _group_offload_block_modules = ["vocoder", "bwe_generator"], so when use_stream=True is used on pipelines with LTX2VocoderWithBWE (such as LTX-2.5 and LTX2DFRPipeline), vocoder weights remain on CPU when its forward runs.
In _apply_group_offloading_block_level, when a model recurses into block_modules (matched_module_groups is empty at the parent level) and also owns root-level parameters/buffers (such as latents_mean / latents_std on LTX VAEs), calling _apply_lazy_group_offloading_hook on the parent registers layer_execution_tracker a second time on child submodules and raises ValueError: Hook with name layer_execution_tracker already exists in the registry. In addition, unmatched_group in _apply_group_offloading_block_level did not forward low_cpu_mem_usage=config.low_cpu_mem_usage.
(LTX2BaseTesterConfig and LTX2DFRPipelineTesterConfig currently work around (1) and (2) in tests via group_offloading_block_level_exclude_modules = ["vae", "audio_vae"].)
Reproduction
importtorchfromdiffusersimportLTX2Pipelinefromdiffusers.pipelines.ltx2.utilsimportDEFAULT_NEGATIVE_PROMPT, DISTILLED_SIGMA_VALUESpipe=LTX2Pipeline.from_pretrained("Lightricks/LTX-2.5-Diffusers", dtype=torch.bfloat16)
pipe.vae.enable_tiling()
pipe.enable_group_offload(
onload_device=torch.device("cuda"),
offload_device=torch.device("cpu"),
offload_type="block_level",
num_blocks_per_group=1,
use_stream=True,
low_cpu_mem_usage=True,
)
video, audio=pipe(
prompt="A red fox walking through a snowy forest at dawn.",
negative_prompt=DEFAULT_NEGATIVE_PROMPT,
width=960,
height=544,
num_frames=121,
frame_rate=24.0,
sigmas=DISTILLED_SIGMA_VALUES,
guidance_scale=1.0,
audio_guidance_scale=1.0,
stg_scale=0.0,
audio_stg_scale=0.0,
modality_scale=1.0,
audio_modality_scale=1.0,
output_type="np",
return_dict=False,
)
Error traceback on main:
RuntimeError: Input type (CUDABFloat16Type) and weight type (CPUBFloat16Type) should be the same
System Info
diffusers: main (f598e6e)
torch: 2.13.0+rocm10.0.0
Python: 3.12
Happy to open a focused PR covering (1)–(4) (and removing the vae / audio_vae test exclusions across the LTX and LTX-2 pipeline suites) once maintainers confirm the scope.
Opened #15345 for this, covering (1)-(4) and removing the vae / audio_vae test exclusions. It includes a before/after run of the reproduction above on LTX-2.5.
Activity
DN6 commentedon Nov 17, 2026
@Shan2L Would you be able to open a PR to address. Feel free to tag me for a review.
Shan2L commentedon Nov 19, 2026
Opened #15345 for this, covering (1)-(4) and removing the
vae/audio_vaetest exclusions. It includes a before/after run of the reproduction above on LTX-2.5.