Add structured prompt upsampling to Ideogram4 by apolinario · Pull Request #13860 · huggingface/diffusers · GitHub
Skip to content

Add structured prompt upsampling to Ideogram4 - #13860

Merged
apolinario merged 11 commits into
mainfrom
ideogram4-prompt-enhancement
Jun 4, 2026
Merged

apolinario merged 11 commits into
mainfrom
ideogram4-prompt-enhancement

Conversation

@apolinario

Copy link
Copy Markdown
Collaborator

Ideogram 4 is trained on a structured JSON caption format and quality degrades noticeably on short free-form prompts. The model strongly recommends keeping prompt upsampling on.

This adds an opt-in on-device upsampler that rewrites the user prompt into Ideogram's native JSON caption schema using the pipeline's own Qwen3-VL encoder grafted with a hosted LM head (diffusers/qwen3-vl-8b-instruct-lm-head). Constrained decoding via outlines (soft dep) enforces the schema. Also wired into the modular pipeline as Ideogram4PromptUpsampleStep.

import torch
from diffusers import Ideogram4Pipeline

pipe = Ideogram4Pipeline.from_pretrained("ideogram-ai/ideogram-4-nf4", torch_dtype=torch.bfloat16)
pipe.to("cuda")

# prompt_upsampling=True rewrites the prompt into Ideogram's structured JSON caption on-device,
# using the pipeline's own Qwen3-VL encoder + a grafted LM head.
# Install `outlines` (pip install outlines) for schema-constrained decoding.
image = pipe(
    "a ginger cat wearing a tiny wizard hat reading a spellbook",
    height=2048,
    width=2048,
    prompt_upsampling=True,
    generator=torch.Generator("cuda").manual_seed(0),
).images[0]
image.save("ideogram4.png")

Stacked on #13859.

@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

@apolinario
apolinario requested a review from dg845 June 3, 2026 17:18
@github-actions github-actions Bot added the models label Jun 3, 2026
Comment thread src/diffusers/pipelines/ideogram4/pipeline_ideogram4.py Outdated
@apolinario
apolinario force-pushed the ideogram4-prompt-enhancement branch from f7d1f05 to 04b197e Compare June 3, 2026 21:48
@github-actions github-actions Bot removed the models label Jun 3, 2026
Base automatically changed from add-ideogram-4 to main June 3, 2026 22:03
@github-actions github-actions Bot added documentation Improvements or additions to documentation models tests labels Jun 3, 2026
Rewrite prompts into Ideogram4's native structured JSON caption before encoding,
opt-in via `prompt_upsampling=True` in `__call__` or the standalone
`upsample_prompt`. Upsampling is driven by a generative `text_encoder`
(`Qwen3VLForConditionalGeneration`, which carries the LM head); the head-less
`Qwen3VLModel` is still supported for plain conditioning, and `upsample_prompt`
raises an instructive error when the encoder cannot generate. Captions are
schema-constrained via `outlines` when installed, and the modular pipeline gains
a matching prompt-enhancer block.
@apolinario
apolinario force-pushed the ideogram4-prompt-enhancement branch from e8ddda1 to 6f06de9 Compare June 3, 2026 23:24
@github-actions github-actions Bot removed documentation Improvements or additions to documentation models tests labels Jun 3, 2026
Drop `graft_lm_head` and drive the modular `Ideogram4PromptUpsampleStep` off a
generative `text_encoder` (`Qwen3VLForConditionalGeneration`), matching the
standard pipeline: guard with `can_generate()` and an instructive error, and
build the outlines logits processor lazily. Updates the copied
`_get_text_encoder_hidden_states` to resolve the decoder for both encoder
classes.
Reflow the Ideogram4 prompt-enhancer docstrings to the 119-col doc-builder
style, and add `# docstyle-ignore` to the functional `CAPTION_SYSTEM_MESSAGE`
and `CAPTION_USER_TEMPLATE` strings so the styler doesn't rewrap them (matching
Flux2's `system_messages.py`).
Add `Ideogram4PromptEnhancerHead`, a small `ModelMixin` holding the Qwen3-VL LM
head, as an optional `prompt_enhancer_head` pipeline component. Upsampling loads
the head via a normal `from_pretrained` (its own repo, or bundled later) instead
of an in-pipeline download, and grafts it onto the shared head-less `text_encoder`
body so no second 8B body is loaded. Both the standard and modular pipelines build
the generative model from `text_encoder` + the head; `upsample_prompt` raises an
instructive error when the head component is absent.
Comment thread src/diffusers/pipelines/ideogram4/pipeline_ideogram4.py Outdated
Comment thread src/diffusers/pipelines/ideogram4/pipeline_ideogram4.py
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
Comment thread src/diffusers/modular_pipelines/ideogram4/encoders.py

@dg845 dg845 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! Left some small comments.

Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
Comment thread src/diffusers/modular_pipelines/ideogram4/encoders.py
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
Comment thread src/diffusers/modular_pipelines/ideogram4/encoders.py Outdated
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Jun 4, 2026
@apolinario
apolinario merged commit 3e83f43 into main Jun 4, 2026
18 of 20 checks passed
DN6 pushed a commit that referenced this pull request Jul 1, 2026
* Add structured prompt upsampling to Ideogram4

Rewrite prompts into Ideogram4's native structured JSON caption before encoding,
opt-in via `prompt_upsampling=True` in `__call__` or the standalone
`upsample_prompt`. Upsampling is driven by a generative `text_encoder`
(`Qwen3VLForConditionalGeneration`, which carries the LM head); the head-less
`Qwen3VLModel` is still supported for plain conditioning, and `upsample_prompt`
raises an instructive error when the encoder cannot generate. Captions are
schema-constrained via `outlines` when installed, and the modular pipeline gains
a matching prompt-enhancer block.

* Remove LM-head grafting; modular block uses a generative text_encoder

Drop `graft_lm_head` and drive the modular `Ideogram4PromptUpsampleStep` off a
generative `text_encoder` (`Qwen3VLForConditionalGeneration`), matching the
standard pipeline: guard with `can_generate()` and an instructive error, and
build the outlines logits processor lazily. Updates the copied
`_get_text_encoder_hidden_states` to resolve the decoder for both encoder
classes.

* Style docstrings with doc-builder; mark prompt strings docstyle-ignore

Reflow the Ideogram4 prompt-enhancer docstrings to the 119-col doc-builder
style, and add `# docstyle-ignore` to the functional `CAPTION_SYSTEM_MESSAGE`
and `CAPTION_USER_TEMPLATE` strings so the styler doesn't rewrap them (matching
Flux2's `system_messages.py`).

* Use an Ideogram4PromptEnhancerHead component for prompt upsampling

Add `Ideogram4PromptEnhancerHead`, a small `ModelMixin` holding the Qwen3-VL LM
head, as an optional `prompt_enhancer_head` pipeline component. Upsampling loads
the head via a normal `from_pretrained` (its own repo, or bundled later) instead
of an in-pipeline download, and grafts it onto the shared head-less `text_encoder`
body so no second 8B body is loaded. Both the standard and modular pipelines build
the generative model from `text_encoder` + the head; `upsample_prompt` raises an
instructive error when the head component is absent.

* Update src/diffusers/pipelines/ideogram4/pipeline_ideogram4.py

Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>

* Update src/diffusers/pipelines/ideogram4/pipeline_ideogram4.py

Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>

* Apply suggestion from @apolinario

* Apply suggestion from @dg845

Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>

* Apply suggestion from @apolinario

* Fix trailing whitespace

* docs: add prompt-upsampling examples (remote API + local head) for Ideogram4

---------

Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation modular-pipelines pipelines size/L PR with diff > 200 LOC utils

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants