Skip to content

enable_group_offload(offload_type="block_level") fails on LTX / LTX-2 / LTX-2.5 pipelines (vae.decode, audio_vae.decode, LTX2VocoderWithBWE, and nested block_modules) #15250

Open

Description

@Shan2L

Describe the bug

Enabling pipeline-level block group offloading (pipe.enable_group_offload(..., offload_type="block_level", num_blocks_per_group=1)) on LTX / LTX-2 / LTX-2.5 pipelines fails at decode time with CPU/GPU device mismatches (and ValueError when use_stream=True is combined with _group_offload_block_modules on models that also hold root-level buffers):

  1. AutoencoderKLLTXVideo and AutoencoderKLLTX2Video do not define _group_offload_block_modules = ["encoder", "decoder"], so encoder and decoder fall into the root unmatched_group whose hook is only triggered on vae.forward(), whereas pipelines invoke vae.decode() / vae.encode() directly.
  2. AutoencoderKLLTX2Audio not only lacks _group_offload_block_modules = ["encoder", "decoder"], its LTX2AudioEncoder / LTX2AudioDecoder stages are bare nn.Module() containers iterated via attribute access (stage.block, stage.upsample) rather than stage(...), bypassing GroupOffloadingHook registered on self.down / self.up entries.
  3. LTX2VocoderWithBWE wraps vocoder and bwe_generator (LTX2Vocoder instances) without _group_offload_block_modules = ["vocoder", "bwe_generator"], so when use_stream=True is used on pipelines with LTX2VocoderWithBWE (such as LTX-2.5 and LTX2DFRPipeline), vocoder weights remain on CPU when its forward runs.
  4. In _apply_group_offloading_block_level, when a model recurses into block_modules (matched_module_groups is empty at the parent level) and also owns root-level parameters/buffers (such as latents_mean / latents_std on LTX VAEs), calling _apply_lazy_group_offloading_hook on the parent registers layer_execution_tracker a second time on child submodules and raises ValueError: Hook with name layer_execution_tracker already exists in the registry. In addition, unmatched_group in _apply_group_offloading_block_level did not forward low_cpu_mem_usage=config.low_cpu_mem_usage.

(LTX2BaseTesterConfig and LTX2DFRPipelineTesterConfig currently work around (1) and (2) in tests via group_offloading_block_level_exclude_modules = ["vae", "audio_vae"].)

Reproduction

import torch
from diffusers import LTX2Pipeline
from diffusers.pipelines.ltx2.utils import DEFAULT_NEGATIVE_PROMPT, DISTILLED_SIGMA_VALUES

pipe = LTX2Pipeline.from_pretrained("Lightricks/LTX-2.5-Diffusers", dtype=torch.bfloat16)
pipe.vae.enable_tiling()
pipe.enable_group_offload(
    onload_device=torch.device("cuda"),
    offload_device=torch.device("cpu"),
    offload_type="block_level",
    num_blocks_per_group=1,
    use_stream=True,
    low_cpu_mem_usage=True,
)

video, audio = pipe(
    prompt="A red fox walking through a snowy forest at dawn.",
    negative_prompt=DEFAULT_NEGATIVE_PROMPT,
    width=960,
    height=544,
    num_frames=121,
    frame_rate=24.0,
    sigmas=DISTILLED_SIGMA_VALUES,
    guidance_scale=1.0,
    audio_guidance_scale=1.0,
    stg_scale=0.0,
    audio_stg_scale=0.0,
    modality_scale=1.0,
    audio_modality_scale=1.0,
    output_type="np",
    return_dict=False,
)

Error traceback on main:

RuntimeError: Input type (CUDABFloat16Type) and weight type (CPUBFloat16Type) should be the same

System Info

  • diffusers: main (f598e6e)
  • torch: 2.13.0+rocm10.0.0
  • Python: 3.12

Happy to open a focused PR covering (1)–(4) (and removing the vae / audio_vae test exclusions across the LTX and LTX-2 pipeline suites) once maintainers confirm the scope.

Activity

  1. DN6 commented on Nov 17, 2026

    @DN6
    Collaborator

    @Shan2L Would you be able to open a PR to address. Feel free to tag me for a review.

  2. Shan2L commented on Nov 19, 2026

    @Shan2L
    ContributorAuthor

    Opened #15345 for this, covering (1)-(4) and removing the vae / audio_vae test exclusions. It includes a before/after run of the reproduction above on LTX-2.5.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions