Merge branch 'main' of https://github.com/huggingface/diffusers · mingyi456/diffusers@55d8034 · GitHub
Skip to content

Commit 55d8034

Browse files
committed
* 'main' of https://github.com/huggingface/diffusers: [tests] remove `test_pipelines_common.py` (huggingface#14669) [chore] remove force_upcast from autoencoder flux2 (huggingface#14946) Configure fixed sampling sigmas on QwenImage21Pipeline (huggingface#14950) [tests] refactor the pipeline-level attention processor tests (huggingface#14701) [modular] Echo team joy future academy jd echo by @Gelercatty (huggingface#14941) Fix: Deprecating Siglip2ImageProcessorFast (huggingface#14781)
2 parents 06e6d81 + 7717798 commit 55d8034

76 files changed

Lines changed: 3550 additions & 4396 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.ai/references/testing.md‎

Lines changed: 1 addition & 1 deletion

‎.github/workflows/nightly_tests.yml‎

Lines changed: 10 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,7 @@ jobs:
5454
5555
- name: Pipeline Tests Artifacts
5656
if: ${{ always() }}
57-
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
57+
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
5858
with:
5959
name: test-pipelines.json
6060
path: reports
@@ -108,7 +108,7 @@ jobs:
108108
cat reports/tests_pipeline_${{ matrix.module }}_cuda_failures_short.txt
109109
- name: Test suite reports artifacts
110110
if: ${{ always() }}
111-
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
111+
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
112112
with:
113113
name: pipeline_${{ matrix.module }}_test_reports
114114
path: reports
@@ -180,7 +180,7 @@ jobs:
180180
181181
- name: Test suite reports artifacts
182182
if: ${{ always() }}
183-
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
183+
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
184184
with:
185185
name: torch_${{ matrix.module }}_cuda_test_reports
186186
path: reports
@@ -225,7 +225,7 @@ jobs:
225225

226226
- name: Test suite reports artifacts
227227
if: ${{ always() }}
228-
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
228+
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
229229
with:
230230
name: torch_compile_test_reports
231231
path: reports
@@ -279,7 +279,7 @@ jobs:
279279
cat reports/tests_big_gpu_torch_cuda_failures_short.txt
280280
- name: Test suite reports artifacts
281281
if: ${{ always() }}
282-
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
282+
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
283283
with:
284284
name: torch_cuda_big_gpu_test_reports
285285
path: reports
@@ -323,7 +323,6 @@ jobs:
323323
pytest -n 1 --max-worker-restart=0 --dist=loadfile \
324324
--make-reports=tests_torch_minimum_version_cuda \
325325
tests/models/test_modeling_common.py \
326-
tests/pipelines/test_pipelines_common.py \
327326
tests/pipelines/test_pipeline_utils.py \
328327
tests/pipelines/test_pipelines.py \
329328
tests/pipelines/test_pipelines_auto.py \
@@ -338,7 +337,7 @@ jobs:
338337
339338
- name: Test suite reports artifacts
340339
if: ${{ always() }}
341-
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
340+
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
342341
with:
343342
name: torch_minimum_version_cuda_test_reports
344343
path: reports
@@ -410,7 +409,7 @@ jobs:
410409
cat reports/tests_${{ matrix.config.backend }}_torch_cuda_failures_short.txt
411410
- name: Test suite reports artifacts
412411
if: ${{ always() }}
413-
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
412+
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
414413
with:
415414
name: torch_cuda_${{ matrix.config.backend }}_reports
416415
path: reports
@@ -463,7 +462,7 @@ jobs:
463462
cat reports/tests_pipeline_level_quant_torch_cuda_failures_short.txt
464463
- name: Test suite reports artifacts
465464
if: ${{ always() }}
466-
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
465+
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
467466
with:
468467
name: torch_cuda_pipeline_level_quant_reports
469468
path: reports
@@ -496,7 +495,7 @@ jobs:
496495
run: mkdir -p combined_reports
497496

498497
- name: Download all test reports
499-
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
498+
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7.0.0
500499
with:
501500
path: artifacts
502501

@@ -522,7 +521,7 @@ jobs:
522521
cat $CONSOLIDATED_REPORT_PATH >> $GITHUB_STEP_SUMMARY
523522
524523
- name: Upload consolidated report
525-
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
524+
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
526525
with:
527526
name: consolidated_test_report
528527
path: ${{ env.CONSOLIDATED_REPORT_PATH }}

‎.github/workflows/pr_tests_gpu.yml‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,9 @@ on:
1212
- "src/diffusers/loaders/lora_base.py"
1313
- "src/diffusers/loaders/lora_pipeline.py"
1414
- "src/diffusers/loaders/peft.py"
15-
- "tests/pipelines/test_pipelines_common.py"
15+
- "tests/pipelines/testing_utils/**.py"
1616
- "tests/models/test_modeling_common.py"
17+
- "tests/models/testing_utils/**.py"
1718
- "examples/**/*.py"
1819
workflow_dispatch:
1920

@@ -94,7 +95,7 @@ jobs:
9495
runs-on:
9596
group: aws-general-8-plus
9697
container:
97-
image: diffusers/diffusers-pytorch-cpu
98+
image: diffusers/diffusers-pytorch-cpu:latest
9899
outputs:
99100
pipeline_test_matrix: ${{ steps.fetch_pipeline_matrix.outputs.pipeline_test_matrix }}
100101
steps:
@@ -135,7 +136,7 @@ jobs:
135136
runs-on:
136137
group: aws-g4dn-2xlarge
137138
container:
138-
image: diffusers/diffusers-pytorch-cuda
139+
image: diffusers/diffusers-pytorch-cuda:latest
139140
options: --shm-size "16gb" --ipc host --gpus all
140141
steps:
141142
- name: Checkout diffusers
@@ -201,7 +202,7 @@ jobs:
201202
runs-on:
202203
group: aws-g4dn-2xlarge
203204
container:
204-
image: diffusers/diffusers-pytorch-cuda
205+
image: diffusers/diffusers-pytorch-cuda:latest
205206
options: --shm-size "16gb" --ipc host --gpus all
206207
defaults:
207208
run:
@@ -273,7 +274,7 @@ jobs:
273274
group: aws-g4dn-2xlarge
274275

275276
container:
276-
image: diffusers/diffusers-pytorch-cuda
277+
image: diffusers/diffusers-pytorch-cuda:latest
277278
options: --gpus all --shm-size "16gb" --ipc host
278279
steps:
279280
- name: Checkout diffusers

‎.github/workflows/release_tests_fast.yml‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -213,7 +213,6 @@ jobs:
213213
pytest -n 1 --max-worker-restart=0 --dist=loadfile \
214214
--make-reports=tests_torch_minimum_cuda \
215215
tests/models/test_modeling_common.py \
216-
tests/pipelines/test_pipelines_common.py \
217216
tests/pipelines/test_pipeline_utils.py \
218217
tests/pipelines/test_pipelines.py \
219218
tests/pipelines/test_pipelines_auto.py \

‎docs/source/en/_toctree.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -575,6 +575,8 @@
575575
title: DreamLite
576576
- local: api/pipelines/easyanimate
577577
title: EasyAnimate
578+
- local: api/pipelines/echo
579+
title: Echo
578580
- local: api/pipelines/ernie_image
579581
title: ERNIE-Image
580582
- local: api/pipelines/flux
Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
<!-- Copyright 2026 The HuggingFace Team. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License. -->
14+
15+
# Echo
16+
17+
[Echo](https://github.com/jd-opensource/JoyAI-Echo) is a long-video generation model. It supports an optional clean
18+
first frame, ordered image/audio memory slots, and a stochastic few-step Distribution Matching Distillation (DMD)
19+
sampler.
20+
The pipeline generates synchronized video and audio.
21+
22+
Echo is implemented as a [`ModularPipeline`] so its text encoding, memory conditioning, stochastic DMD denoising, and
23+
decoding blocks can be run as a complete workflow or composed independently.
24+
25+
## Inference
26+
27+
Load the official [jdopensource/JoyAI-Echo](https://huggingface.co/jdopensource/JoyAI-Echo) checkpoint directly.
28+
It uses the text encoder and tokenizer from [google/gemma-3-12b-it](https://huggingface.co/google/gemma-3-12b-it).
29+
The Gemma repository is gated, so accept its license and authenticate with Hugging Face before loading the pipeline.
30+
31+
The released model uses 241 frames in its long-video example. The video RoPE coordinates remain at the training rate
32+
of 24 fps, independently of the output container rate.
33+
34+
```py
35+
import torch
36+
import torchaudio
37+
from PIL import Image
38+
39+
from diffusers import ComponentsManager, ModularPipeline
40+
from diffusers.utils import encode_video
41+
42+
manager = ComponentsManager()
43+
pipe = ModularPipeline.from_pretrained("jdopensource/JoyAI-Echo", components_manager=manager)
44+
pipe.load_components(dtype={"default": torch.bfloat16, "audio_vae": torch.float32})
45+
manager.enable_auto_cpu_offload(device="cuda")
46+
pipe.vae.enable_tiling()
47+
48+
first_frame = Image.open("first_frame.png").convert("RGB")
49+
memory_images = [Image.open(path).convert("RGB") for path in ["memory_0.png", "memory_1.png"]]
50+
memory_audio_with_rates = [torchaudio.load(path) for path in ["memory_0.wav", "memory_1.wav"]]
51+
memory_audio = [waveform for waveform, _ in memory_audio_with_rates]
52+
memory_audio_rates = [sample_rate for _, sample_rate in memory_audio_with_rates]
53+
54+
output = pipe(
55+
prompt="A cinematic dialogue scene in a quiet cafe.",
56+
image=first_frame,
57+
memory_images=memory_images,
58+
memory_audio_waveforms=memory_audio,
59+
memory_audio_sample_rates=memory_audio_rates,
60+
width=1280,
61+
height=736,
62+
num_frames=241,
63+
frame_rate=25.0,
64+
model_frame_rate=24.0,
65+
generator=torch.Generator(device="cuda").manual_seed(42),
66+
output_type="np",
67+
output=["videos", "audio"],
68+
)
69+
70+
encode_video(
71+
output["videos"][0],
72+
fps=25,
73+
audio=output["audio"][0].float().cpu(),
74+
audio_sample_rate=pipe.vocoder.config.output_sampling_rate,
75+
output_path="echo.mp4",
76+
)
77+
```
78+
79+
The default DMD sigma schedule is the released eight-step schedule. It predicts `x0` at every step and re-noises with
80+
fresh Gaussian noise at the next sigma, so a seeded `torch.Generator` controls both the initial noise and all
81+
intermediate re-noising.
82+
83+
Raw audio-memory encoding requires `torchaudio`. For reference parity, keep `audio_vae` in FP32 as shown above.
84+
Modular workflows can cache the VAE encoder's normalized, unpacked tensors. Video latents have shape
85+
`(batch, channels, frames, height, width)` and audio latents have shape `(batch, channels, time, mel_bins)`.
86+
The core `denoise` block packs those tensors, expands conditioning for `num_videos_per_prompt`, and unpacks
87+
denoised outputs back to the same VAE form. Pass initial `latents` and `audio_latents` in that unpacked form too.
88+
Decoders take normalized VAE tensors and denormalize immediately before
89+
decoding. `output=["latents", "audio_latents"]` returns normalized VAE tensors. With `output_type="latent"` and
90+
`output=["videos", "audio"]`, you get the denormalized VAE tensors without decoding.
91+
92+
## EchoModularPipeline
93+
94+
[[autodoc]] EchoModularPipeline
95+
96+
## EchoBlocks
97+
98+
[[autodoc]] EchoBlocks

‎docs/source/en/api/pipelines/qwenimage21.md‎

Lines changed: 20 additions & 0 deletions

0 commit comments

Comments
 (0)