Add Ideogram 4 by apolinario · Pull Request #13859 · huggingface/diffusers · GitHub
Skip to content

Add Ideogram 4 - #13859

Merged
yiyixuxu merged 2 commits into
mainfrom
add-ideogram-4
Jun 3, 2026
Merged

yiyixuxu merged 2 commits into
mainfrom
add-ideogram-4

Conversation

@apolinario

@apolinario apolinario commented Jun 3, 2026 •

Copy link
Copy Markdown
Collaborator

Adds the Ideogram 4 text-to-image model: transformer, standard pipeline, modular pipeline, docs, and tests.

Checkpoint: ideogram-ai/ideogram-4-nf4 (gated, please accept the gate before using with diffusers)

diffusers — remote prompt upsampling

For the best possible results, use the Ideogram prompt upsampling.

import json, os, torch, requests
from diffusers import Ideogram4Pipeline

pipe = Ideogram4Pipeline.from_pretrained(
    "ideogram-ai/ideogram-4-nf4",
    torch_dtype=torch.bfloat16,
    token=os.environ["HF_TOKEN"], # or: token="hf_xxxxxxxxx", token is needed as the repo is gated
).to("cuda")

# Expand the prompt into a structured JSON caption with Ideogram's free hosted magic-prompt API.
# Get a key at https://developer.ideogram.ai/  (set IDEOGRAM_API_KEY).
resp = requests.post(
    "https://api.ideogram.ai/v1/ideogram-v4/magic-prompt",
    headers={"Api-Key": "your_ideogram_api_key"},
    json={"text_prompt": "a ginger cat wearing a tiny wizard hat reading a spellbook", "aspect_ratio": "1x1"},
).json()
caption = json.dumps(resp["json_prompt"])  # already a structured JSON caption

# Pass the caption straight to the pipeline (no prompt_upsampling — it's already upsampled).
image = pipe(
    caption,
    height=1024,  # model supports up to 2048
    width=1024,   # model supports up to 2048
    generator=torch.Generator("cuda").manual_seed(0),
).images[0]
image.save("ideogram4.png")

diffusers — local prompt upsampling

For a full-local experience, diffusers ships a prompt_upsampling that utilizes the same Qwen3-VL-8B model as the text-encoder for the upsampling. Expect a quality decrease compared to remote prompt upsampling.

pip install outlines  # to force the json structure
import torch
from diffusers import Ideogram4Pipeline

pipe = Ideogram4Pipeline.from_pretrained(
    "ideogram-ai/ideogram-4-nf4",
    torch_dtype=torch.bfloat16,
    token=os.environ["HF_TOKEN"], # or: token="hf_xxxxxxxxx", token is needed as the repo is gated
).to("cuda")

# prompt_upsampling=True rewrites the prompt into Ideogram's structured JSON caption locally on-device.
image = pipe(
    "a ginger cat wearing a tiny wizard hat reading a spellbook",
    height=1024,  # model supports up to 2048
    width=1024,   # model supports up to 2048
    prompt_upsampling=True,
    generator=torch.Generator("cuda").manual_seed(0),
).images[0]
image.save("ideogram4.png")

Adds the Ideogram 4 text-to-image model: transformer, standard pipeline,
modular pipeline, docs, and tests.

Checkpoint: ideogram-ai/ideogram-4-nf4

Co-Authored-By: YiYi Xu <yiyi@huggingface.co>
@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

Replace the fused `qkv`/`o` linears with canonical `to_q`/`to_k`/`to_v`/`to_out`
projections, matching the standard diffusers attention layout and the split
checkpoint format. Mathematically equivalent to the fused form (q/k/v are
contiguous row-slices of the fused weight). Drops the now-inapplicable
fuse/unfuse overrides.

@yiyixuxu yiyixuxu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!!

@yiyixuxu
yiyixuxu merged commit 9b0818c into main Jun 3, 2026
19 of 20 checks passed
@yiyixuxu
yiyixuxu deleted the add-ideogram-4 branch June 3, 2026 22:03
apolinario added a commit to apolinario/ideogram4 that referenced this pull request Jun 4, 2026
Replace the fused attention.qkv / attention.o linear layers with split
to_q/to_k/to_v and to_out, mirroring the diffusers loader change in
huggingface/diffusers#13859 (commit fbe4750). q/k/v are contiguous
row-slices of the old fused weight, so this is mathematically identical.

Loads the split-format checkpoints published for
ideogram-ai/ideogram-4-nf4 and ideogram-ai/ideogram-4-fp8.
DN6 pushed a commit that referenced this pull request Jul 1, 2026
* Add Ideogram 4

Adds the Ideogram 4 text-to-image model: transformer, standard pipeline,
modular pipeline, docs, and tests.

Checkpoint: ideogram-ai/ideogram-4-nf4

Co-Authored-By: YiYi Xu <yiyi@huggingface.co>

* Use split q/k/v projections in Ideogram4 attention

Replace the fused `qkv`/`o` linears with canonical `to_q`/`to_k`/`to_v`/`to_out`
projections, matching the standard diffusers attention layout and the split
checkpoint format. Mathematically equivalent to the fused form (q/k/v are
contiguous row-slices of the fused weight). Drops the now-inapplicable
fuse/unfuse overrides.

---------

Co-authored-by: Jin <jin.li@ideogram.ai>
Co-authored-by: YiYi Xu <yiyi@huggingface.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation models modular-pipelines pipelines size/L PR with diff > 200 LOC tests utils

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants