Skip to content

[enhancement]: Prompt enhancer as a standalone model type (ERNIE-Image) #9738

Description

@Pfannkuchensack

Is there an existing issue for this?

  • I have searched the existing issues

What should this feature add?

Give ModelType.PromptEnhancer a config and a loader so ERNIE-Image's prompt enhancer installs as its own model.
Today the type exists without either, and the enhancer is only usable from inside the Diffusers pipeline, so a
single-file or GGUF ERNIE-Image install has no prompt enhancer.

Notes from an earlier investigation:

  • The enhancer has the same geometry as the Ministral 3B encoder; the distinguishing signal is the missing Pixtral
    vision tower (only the encoder carries it). Today the encoder probe refuses the enhancer by name.
  • Ministral3ForCausalLM's lm_head is tied and absent from the released file; it must be tied after loading,
    or a randomly initialised head generates nonsense without failing.
  • apply_chat_template works on the Tekken tokenizer backend.
  • Metadata: keep the typed prompt as positive_prompt and store the enhanced text in a separate field.

Alternatives

Use the Diffusers pipeline, which carries the enhancer.

Additional Content

Needs an optional prompt_enhancer_model on the ERNIE-Image loader node and a Generate slot.

The full matrix of what loads today is in #9726 (Model Format Support page).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions