Main to clipdrop-main (#12) * Fix broken group offloading with block_level for models with standalone layers (#12692) * fix: group offloading to support standalone computational layers in block-level offloading * test: for models with standalone and deeply nested layers in block-level offloading * feat: support for block-level offloading in group offloading config * fix: group offload block modules to AutoencoderKL and AutoencoderKLWan * fix: update group offloading tests to use AutoencoderKL and adjust input dimensions * refactor: streamline block offloading logic * Apply style fixes * update tests * update * fix for failing tests * clean up * revert to use skip_keys * clean up --------- Co-authored-by: github-actions[bot] Co-authored-by: Dhruv Nair * [Docs] Add Z-Image docs (#12775) * initial * toctree * fix * apply review and fix * Update docs/source/en/api/pipelines/z_image.md Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Update docs/source/en/api/pipelines/z_image.md Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Update docs/source/en/api/pipelines/z_image.md Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> --------- Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * move kandisnky docs. * [docs] minor fixes to kandinsky docs (#12797) up * Improve docstrings and type hints in scheduling_deis_multistep.py (#12796) * feat: Add `flow_prediction` to `prediction_type`, introduce `use_flow_sigmas`, `flow_shift`, `use_dynamic_shifting`, and `time_shift_type` parameters, and refine type hints for various arguments. * style: reformat argument wrapping in `_convert_to_beta` and `index_for_timestep` method signatures. * [Feat] TaylorSeer Cache (#12648) * init taylor_seer cache * make compatible with any tuple size returned * use logger for printing, add warmup feature * still update in warmup steps * refractor, add docs * add configurable cache, skip compute module * allow special cache ids only * add stop_predicts (cooldown) * update docs * apply ruff * update to handle multple calls per timestep * refractor to use state manager * fix format & doc * chores: naming, remove redundancy * add docs * quality & style * fix taylor precision * Apply style fixes * add tests * Apply style fixes * Remove TaylorSeerCacheTesterMixin from flux2 tests * rename identifiers, use more expressive taylor predict loop * torch compile compatible * Apply style fixes * Update src/diffusers/hooks/taylorseer_cache.py Co-authored-by: Dhruv Nair * update docs * make fix-copies * fix example usage. * remove tests on flux kontext --------- Co-authored-by: toilaluan Co-authored-by: github-actions[bot] Co-authored-by: Dhruv Nair Co-authored-by: Sayak Paul * Update the TensorRT-ModelOPT to Nvidia-ModelOPT (#12793) Update the naming Co-authored-by: Sayak Paul * add post init for safty checker (#12794) * add post init for safty checker Signed-off-by: jiqing-feng * check transformers version before post init Signed-off-by: jiqing-feng * Apply style fixes --------- Signed-off-by: jiqing-feng Co-authored-by: github-actions[bot] * [HunyuanVideo1.5] support step-distilled (#12802) * support step-distilled * style * Add ZImageImg2ImgPipeline (#12751) * Add ZImageImg2ImgPipeline Updated the pipeline structure to include ZImageImg2ImgPipeline alongside ZImagePipeline. Implemented the ZImageImg2ImgPipeline class for image-to-image transformations, including necessary methods for encoding prompts, preparing latents, and denoising. Enhanced the auto_pipeline to map the new ZImageImg2ImgPipeline for image generation tasks. Added unit tests for ZImageImg2ImgPipeline to ensure functionality and performance. Updated dummy objects to include ZImageImg2ImgPipeline for testing purposes. * Address review comments for ZImageImg2ImgPipeline - Add `# Copied from` annotations to encode_prompt and _encode_prompt - Add ZImagePipeline to auto_pipeline.py for AutoPipeline support * Add ZImage pipeline documentation --------- Co-authored-by: YiYi Xu Co-authored-by: Álvaro Somoza * [PRX] Improve model compilation (#12787) * Reimplement img2seq & seq2img in PRX to enable ONNX build without Col2Im (incompatible with TensorRT). * Apply style fixes --------- Co-authored-by: github-actions[bot] Co-authored-by: Sayak Paul * Improve docstrings and type hints in scheduling_dpmsolver_singlestep.py (#12798) feat: add flow sigmas, dynamic shifting, and refine type hints in DPMSolverSinglestepScheduler * [Modular]z-image (#12808) * initiL * up up * fix: z_image -> z-image * style * copy * fix more * some docstring fix * Fix Qwen Edit Plus modular for multi-image input (#12601) * try to fix qwen edit plus multi images (modular) * up * up * test * up * up * [WIP] Add Flux2 modular (#12763) * update * update * update * update * update * update * update * update * update * update * [docs] improve distributed inference cp docs. (#12810) * improve distributed inference cp docs. * Apply suggestions from code review Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> --------- Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * post release 0.36.0 (#12804) * post release 0.36.0 * Apply style fixes --------- Co-authored-by: github-actions[bot] * Update distributed_inference.md to correct syntax (#12827) * [lora] Remove lora docs unneeded and add " # Copied from ..." (#12824) * remove unneeded docs on load_lora_weights(). * remove more. * up[ * up * up * support CP in native flash attention (#12829) Signed-off-by: Wang, Yi Co-authored-by: Sayak Paul * [qwen-image] edit 2511 support (#12839) * [qwen-image] edit 2511 support * Apply style fixes --------- Co-authored-by: github-actions[bot] * fix pytest tests/pipelines/pixart_sigma/test_pixart.py::PixArtSigmaPi… (#12842) fix pytest tests/pipelines/pixart_sigma/test_pixart.py::PixArtSigmaPipelineIntegrationTests::test_pixart_512 in xpu Signed-off-by: Wang, Yi Co-authored-by: Sayak Paul * Support for control-lora (#10686) * run control-lora on diffusers * cannot load lora adapter * test * 1 * add control-lora * 1 * 1 * 1 * fix PeftAdapterMixin * fix module_to_save bug * delete json print * resolve conflits * merged but bug * change peft.py * 1 * delete state_dict print * fix alpha * Create control_lora.py * Add files via upload * rename * no need modify as peft updated * add doc * fix code style * styling isn't that hard 😉 * empty --------- Co-authored-by: Sayak Paul * Add support for LongCat-Image (#12828) * Add LongCat-Image * Update src/diffusers/models/transformers/transformer_longcat_image.py Co-authored-by: YiYi Xu * Update src/diffusers/models/transformers/transformer_longcat_image.py Co-authored-by: YiYi Xu * Update src/diffusers/models/transformers/transformer_longcat_image.py Co-authored-by: YiYi Xu * Update src/diffusers/pipelines/longcat_image/pipeline_longcat_image.py Co-authored-by: YiYi Xu * Update src/diffusers/pipelines/longcat_image/pipeline_longcat_image.py Co-authored-by: YiYi Xu * Update src/diffusers/pipelines/longcat_image/pipeline_longcat_image.py Co-authored-by: YiYi Xu * Update src/diffusers/models/transformers/transformer_longcat_image.py Co-authored-by: YiYi Xu * Update src/diffusers/pipelines/longcat_image/pipeline_longcat_image.py Co-authored-by: YiYi Xu * fix code * add doc * Update src/diffusers/pipelines/longcat_image/pipeline_longcat_image_edit.py Co-authored-by: YiYi Xu * Update src/diffusers/pipelines/longcat_image/pipeline_longcat_image_edit.py Co-authored-by: YiYi Xu * Update src/diffusers/pipelines/longcat_image/pipeline_longcat_image.py Co-authored-by: YiYi Xu * Update src/diffusers/pipelines/longcat_image/pipeline_longcat_image.py Co-authored-by: YiYi Xu * Update src/diffusers/pipelines/longcat_image/pipeline_longcat_image.py Co-authored-by: YiYi Xu * Update src/diffusers/pipelines/longcat_image/pipeline_longcat_image.py Co-authored-by: YiYi Xu * fix code & mask style & fix-copies * Apply style fixes * fix single input rewrite error --------- Co-authored-by: YiYi Xu Co-authored-by: github-actions[bot] Co-authored-by: hadoop-imagen * fix the prefix_token_len bug (#12845) * extend TorchAoTest::test_model_memory_usage to other platform (#12768) * extend TorchAoTest::test_model_memory_usage to other platform Signe-off-by: Wang, Yi * add some comments Signed-off-by: Wang, Yi --------- Signed-off-by: Wang, Yi * Qwen Image Layered Support (#12853) * [qwen-image] qwen image layered support * [qwen-image] update doc * [qwen-image] fix pr comments * Apply style fixes * make fix-copies --------- Co-authored-by: github-actions[bot] Co-authored-by: Sayak Paul * Z-Image-Turbo ControlNet (#12792) * init --------- Co-authored-by: github-actions[bot] * Cosmos Predict2.5 Base: inference pipeline, scheduler & chkpt conversion (#12852) * cosmos predict2.5 base: convert chkpt & pipeline - New scheduler: scheduling_flow_unipc_multistep.py - Changes to TransformerCosmos for text embeddings via crossattn_proj * scheduler cleanup * simplify inference pipeline * cleanup scheduler + tests * Basic tests for flow unipc * working b2b inference * Rename everything * Tests for pipeline present, but not working (predict2 also not working) * docstring update * wrapper pipelines + make style * remove unnecessary files * UniPCMultistep: support use_karras_sigmas=True and use_flow_sigmas=True * use UniPCMultistepScheduler + fix tests for pipeline * Remove FlowUniPCMultistepScheduler * UniPCMultistepScheduler for use_flow_sigmas=True & use_karras_sigmas=True * num_inference_steps=36 due to bug in scheduler used by predict2.5 * Address comments * make style + make fix-copies * fix tests + remove references to old pipelines * address comments * add revision in from_pretrained call * fix tests * more update in modular (#12560) * move node registry to mellon * up * fix * modula rpipeline update: filter out none for input_names, fix default blocks for pipe.init() and allow user pass additional kwargs_type in a dict * qwen modular refactor, unpack before decode * update mellon node config, adding* to required_inputs and required_model_inputs * modularpipeline.from_pretrained: error out if no config found * add a component_names property to modular blocks to be consistent! * flux image_encoder -> vae_encoder * controlnet_bundle * refator MellonNodeConfig MellonPipelineConfig * refactor & simplify mellon utils * vae_image_encoder -> vae_encoder * mellon config save keep key order * style + copies * add kwargs input for zimage * Feature: Add Mambo-G Guidance as Guider (#12862) * Feature: Add Mambo-G Guidance to Qwen-Image Pipeline * change to guider implementation * fix copied code residual * Update src/diffusers/guiders/magnitude_aware_guidance.py * Apply style fixes --------- Co-authored-by: Pscgylotti Co-authored-by: YiYi Xu Co-authored-by: github-actions[bot] * Add `OvisImagePipeline` in `AUTO_TEXT2IMAGE_PIPELINES_MAPPING` (#12876) * Cosmos Predict2.5 14b Conversion (#12863) 14b conversion * Use `T5Tokenizer` instead of `MT5Tokenizer` (removed in Transformers v5.0+) (#12877) Use `T5Tokenizer` instead of `MT5Tokenizer` Given that the `MT5Tokenizer` in `transformers` is just a "re-export" of `T5Tokenizer` as per https://github.com/huggingface/transformers/blob/v4.57.3/src/transformers/models/mt5/tokenization_mt5.py )on latest available stable Transformers i.e., v4.57.3), this commit updates the imports to point to `T5Tokenizer` instead, so that those still work with Transformers v5.0.0rc0 onwards. * Add z-image-omni-base implementation (#12857) * Add z-image-omni-base implementation * Merged into one transformer for Z-Image. * Fix bugs for controlnet after merging the main branch new feature. * Fix for auto_pipeline, Add Styling. * Refactor noise handling and modulation - Add select_per_token function for per-token value selection - Separate adaptive modulation logic - Cleanify t_noisy/clean variable naming - Move image_noise_mask handler from forward to pipeline * Styling & Formatting. * Rewrite code with more non-forward func & clean forward. 1.Change to one forward with shorter code with omni code (None). 2.Split out non-forward funcs: _build_unified_sequence, _prepare_sequence, patchify, pad. * Styling & Formatting. * Manual check fix-copies in controlnet, Add select_per_token, _patchify_image, _pad_with_ids; Styling. * Add Import in pipeline __init__.py. --------- Co-authored-by: Jerry Qilong Wu Co-authored-by: YiYi Xu * fix torchao quantizer for new torchao versions (#12901) * fix torchao quantizer for new torchao versions Summary: `torchao==0.16.0` (not yet released) has some bc-breaking changes, this PR fixes the diffusers repo with those changes. Specifics on the changes: 1. `UInt4Tensor` is removed: https://github.com/pytorch/ao/pull/3536 2. old float8 tensors v1 are removed: https://github.com/pytorch/ao/pull/3510 In this PR: 1. move the logger variable up (not sure why it was in the middle of the file before) to get better error messages 2. gate the old torchao objects by torchao version Test Plan: import diffusers objects with new versions of torchao works: ```bash > python -c "import torchao; print(torchao.__version__); from diffusers import StableDiffusionPipeline" 0.16.0.dev20251229+cu129 ``` Reviewers: Subscribers: Tasks: Tags: * Apply style fixes --------- Co-authored-by: github-actions[bot] * fix Qwen Image Transformer single file loading mapping function to be consistent with other loader APIs (#12894) fix Qwen single file loading to be consistent with other loader API * Z-Image-Turbo from_single_file fix (#12888) * chore: fix dev version in setup.py (#12904) * Community Pipeline: Add z-image differential img2img (#12882) * Community Pipeline: Add z-image differential img2img * add pipeline for z-image differential img2img diffusion examples : run make style , make quality, and fix white spaces in example doc string. --------- Co-authored-by: r4inm4ker * Fix typo in src/diffusers/pipelines/cosmos/pipeline_cosmos2_5_predict.py (#12914) * Fix wan 2.1 i2v context parallel (#12909) * fix wan 2.1 i2v context parallel * fix wan 2.1 i2v context parallel * fix wan 2.1 i2v context parallel * format * fix the use of device_map in CP docs (#12902) up * [core] remove unneeded autoencoder methods when subclassing from `AutoencoderMixin` (#12873) up * Detect 2.0 vs 2.1 ZImageControlNetModel (#12861) * Detect 2.0 vs 2.1 ZImageControlNetModel * Possibility of control_noise_refiner being removed * Refactor environment variable assignments in workflow (#12916) * Add codeQL workflow (#12917) Updated CodeQL workflow to use reusable workflow from Hugging Face and simplified language matrix. * Delete .github/workflows/codeql.yml * CodeQL workflow for security analysis * Check for attention mask in backends that don't support it (#12892) * check attention mask * Apply style fixes * bugfix --------- Co-authored-by: github-actions[bot] Co-authored-by: Sayak Paul * [Flux.1] improve pos embed for ascend npu by computing on npu (#12897) * [Flux.1] improve pos embed for ascend npu by setting it back to npu computation. * [Flux.2] improve pos embed for ascend npu by setting it back to npu computation. * [LongCat-Image] improve pos embed for ascend npu by setting it back to npu computation. * [Ovis-Image] improve pos embed for ascend npu by setting it back to npu computation. * Remove unused import of is_torch_npu_available --------- Co-authored-by: zhangtao * LTX Video 0.9.8 long multi prompt (#12614) * LTX Video 0.9.8 long multi prompt * Further align comfyui - Added the “LTXEulerAncestralRFScheduler” scheduler, aligned with [sample_euler_ancestral_RF](https://github.com/comfyanonymous/ComfyUI/blob/7d6103325e1c97aa54f963253e3e7f1d6da6947f/comfy/k_diffusion/sampling.py#L234) - Updated the LTXI2VLongMultiPromptPipeline.from_pretrained() method: - Now uses LTXEulerAncestralRFScheduler by default, for better compatibility with the ComfyUI LTXV workflow. - Changed the default value of cond_strength from 1.0 to 0.5, aligning with ComfyUI’s default. - Optimized cross-window overlap blending: moved the latent-space guidance injection to before the UNet and after each step, aligned with[KSamplerX0Inpaint]([ComfyUI/comfy/samplers.py at master · comfyanonymous/ComfyUI](https://github.com/comfyanonymous/ComfyUI/blob/master/comfy/samplers.py#L391)) - Adjusted the default value of skip_steps_sigma_threshold to 1. * align with diffusers contribute rule * Add new pipelines and update imports * Enhance LTXI2VLongMultiPromptPipeline with noise rescaling Refactor LTXI2VLongMultiPromptPipeline to improve documentation and add noise rescaling functionality. * Clean up comments in scheduling_ltx_euler_ancestral_rf.py Removed design notes and limitations from the implementation. * Enhance video generation example with scheduler Updated LTXI2VLongMultiPromptPipeline example to include LTXEulerAncestralRFScheduler for ComfyUI parity. * clean up * style * copies * import ltx scheduler * copies * fix * fix more * up up * up up up * up upup * Apply suggestions from code review * Update docs/source/en/api/pipelines/ltx_video.md * Update docs/source/en/api/pipelines/ltx_video.md --------- Co-authored-by: yiyixuxu * Add FSDP option for Flux2 (#12860) * Add FSDP option for Flux2 * Apply style fixes * Add FSDP option for Flux2 * Add FSDP option for Flux2 * Add FSDP option for Flux2 * Add FSDP option for Flux2 * Add FSDP option for Flux2 * Update examples/dreambooth/README_flux2.md * guard accelerate import. --------- Co-authored-by: Sayak Paul Co-authored-by: github-actions[bot] * Add transformer cache context for SkyReels-V2 pipelines & Update docs (#12837) * feat: Add transformer cache context for conditional and unconditional predictions for skyreels-v2 pipes. * docs: Remove SkyReels-V2 FLF2V model link and add contributor attribution. * [docs] fix torchao typo. (#12883) fix torchao typo. * Update wan.md to remove unneeded hfoptions (#12890) * Improve docstrings and type hints in scheduling_edm_euler.py (#12871) * docs: add comprehensive docstrings and refine type hints for EDM scheduler methods and config parameters. * refactor: Add type hints to DPM-Solver scheduler methods. * [Modular] Video for Mellon (#12924) num_frames and videos * Add LTX 2.0 Video Pipelines (#12915) * Initial LTX 2.0 transformer implementation * Add tests for LTX 2 transformer model * Get LTX 2 transformer tests working * Rename LTX 2 compile test class to have LTX2 * Remove RoPE debug print statements * Get LTX 2 transformer compile tests passing * Fix LTX 2 transformer shape errors * Initial script to convert LTX 2 transformer to diffusers * Add more LTX 2 transformer audio arguments * Allow LTX 2 transformer to be loaded from local path for conversion * Improve dummy inputs and add test for LTX 2 transformer consistency * Fix LTX 2 transformer bugs so consistency test passes * Initial implementation of LTX 2.0 video VAE * Explicitly specify temporal and spatial VAE scale factors when converting * Add initial LTX 2.0 video VAE tests * Add initial LTX 2.0 video VAE tests (part 2) * Get diffusers implementation on par with official LTX 2.0 video VAE implementation * Initial LTX 2.0 vocoder implementation * Use RMSNorm implementation closer to original for LTX 2.0 video VAE * start audio decoder. * init registration. * up * simplify and clean up * up * Initial LTX 2.0 text encoder implementation * Rough initial LTX 2.0 pipeline implementation * up * up * up * up * Add imports for LTX 2.0 Audio VAE * Conversion script for LTX 2.0 Audio VAE Decoder * Add Audio VAE logic to T2V pipeline * Duplicate scheduler for audio latents * Support num_videos_per_prompt for prompt embeddings * LTX 2.0 scheduler and full pipeline conversion * Add script to test full LTX2Pipeline T2V inference * Fix pipeline return bugs * Add LTX 2 text encoder and vocoder to ltx2 subdirectory __init__ * Fix more bugs in LTX2Pipeline.__call__ * Improve CPU offload support * Fix pipeline audio VAE decoding dtype bug * Fix video shape error in full pipeline test script * Get LTX 2 T2V pipeline to produce reasonable outputs * Make LTX 2.0 scheduler more consistent with original code * Fix typo when applying scheduler fix in T2V inference script * Refactor Audio VAE to be simpler and remove helpers (#7) * remove resolve causality axes stuff. * remove a bunch of helpers. * remove adjust output shape helper. * remove the use of audiolatentshape. * move normalization and patchify out of pipeline. * fix * up * up * Remove unpatchify and patchify ops before audio latents denormalization (#9) --------- Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> * Add support for I2V (#8) * start i2v. * up * up * up * up * up * remove uniform strategy code. * remove unneeded code. * Denormalize audio latents in I2V pipeline (analogous to T2V change) (#11) * test i2v. * Move Video and Audio Text Encoder Connectors to Transformer (#12) * Denormalize audio latents in I2V pipeline (analogous to T2V change) * Initial refactor to put video and audio text encoder connectors in transformer * Get LTX 2 transformer tests working after connector refactor * precompute run_connectors,. * fixes * Address review comments * Calculate RoPE double precisions freqs using torch instead of np * Further simplify LTX 2 RoPE freq calc * Make connectors a separate module (#18) * remove text_encoder.py * address yiyi's comments. * up * up * up * up --------- Co-authored-by: sayakpaul * up (#19) * address initial feedback from lightricks team (#16) * cross_attn_timestep_scale_multiplier to 1000 * implement split rope type. * up * propagate rope_type to rope embed classes as well. * up * When using split RoPE, make sure that the output dtype is same as input dtype * Fix apply split RoPE shape error when reshaping x to 4D * Add export_utils file for exporting LTX 2.0 videos with audio * Tests for T2V and I2V (#6) * add ltx2 pipeline tests. * up * up * up * up * remove content * style * Denormalize audio latents in I2V pipeline (analogous to T2V change) * Initial refactor to put video and audio text encoder connectors in transformer * Get LTX 2 transformer tests working after connector refactor * up * up * i2v tests. * up * Address review comments * Calculate RoPE double precisions freqs using torch instead of np * Further simplify LTX 2 RoPE freq calc * revert unneded changes. * up * up * update to split style rope. * up --------- Co-authored-by: Daniel Gu * up * use export util funcs. * Point original checkpoint to LTX 2.0 official checkpoint * Allow the I2V pipeline to accept image URLs * make style and make quality * remove function map. * remove args. * update docs. * update doc entries. * disable ltx2_consistency test * Simplify LTX 2 RoPE forward by removing coords is None logic * make style and make quality * Support LTX 2.0 audio VAE encoder * Apply suggestions from code review Co-authored-by: Sayak Paul * Remove print statement in audio VAE * up * Fix bug when calculating audio RoPE coords * Ltx 2 latent upsample pipeline (#12922) * Initial implementation of LTX 2.0 latent upsampling pipeline * Add new LTX 2.0 spatial latent upsampler logic * Add test script for LTX 2.0 latent upsampling * Add option to enable VAE tiling in upsampling test script * Get latent upsampler working with video latents * Fix typo in BlurDownsample * Add latent upsample pipeline docstring and example * Remove deprecated pipeline VAE slicing/tiling methods * make style and make quality * When returning latents, return unpacked and denormalized latents for T2V and I2V * Add model_cpu_offload_seq for latent upsampling pipeline --------- Co-authored-by: Daniel Gu * Fix latent upsampler filename in LTX 2 conversion script * Add latent upsample pipeline to LTX 2 docs * Add dummy objects for LTX 2 latent upsample pipeline * Set default FPS to official LTX 2 ckpt default of 24.0 * Set default CFG scale to official LTX 2 ckpt default of 4.0 * Update LTX 2 pipeline example docstrings * make style and make quality * Remove LTX 2 test scripts * Fix LTX 2 upsample pipeline example docstring * Add logic to convert and save a LTX 2 upsampling pipeline * Document LTX2VideoTransformer3DModel forward pass --------- Co-authored-by: sayakpaul * Add environment variables to checkout step (#12927) * Improve docstrings and type hints in scheduling_consistency_decoder.py (#12928) docs: improve docstring scheduling_consistency_decoder.py * Fix: Remove hardcoded CUDA autocast in Kandinsky 5 to fix import warning (#12814) * Fix: Remove hardcoded CUDA autocast in Kandinsky 5 to fix import warning * Apply style fixes * Fix: Remove import-time autocast in Kandinsky to prevent warnings - Removed @torch.autocast decorator from Kandinsky classes. - Implemented manual F.linear casting to ensure numerical parity with FP32. - Verified bit-exact output matches main branch. Co-authored-by: hlky * Used _keep_in_fp32_modules to align with standards --------- Co-authored-by: github-actions[bot] Co-authored-by: hlky * Upgrade GitHub Actions for Node 24 compatibility (#12865) Signed-off-by: Salman Muin Kayser Chishti <13schishti@gmail.com> * fix the warning torch_dtype is deprecated (#12841) * fix the warning torch_dtype is deprecated * Add transformers version check (>= 4.56.0) for dtype parameter * Fix linting errors * [NPU] npu attention enable ulysses (#12919) * npu attention enable ulysses * clean the format * register _native_npu_attention to _supports_context_parallel Signed-off-by: yyt * change npu_fusion_attention's input_layout to BSND to eliminate redundant transpose Signed-off-by: yyt * Update format --------- Signed-off-by: yyt Co-authored-by: Sayak Paul * Torchao floatx version guard (#12923) * Adding torchao version guard for floatx usage Summary: TorchAO removing floatx support, added version guard in quantization_config.py * Adding torchao version guard for floatx usage Summary: TorchAO removing floatx support, added version guard in quantization_config.py Altered tests in test_torchao.py to version guard floatx Created new test to verify version guard of floatx support * Adding torchao version guard for floatx usage Summary: TorchAO removing floatx support, added version guard in quantization_config.py Altered tests in test_torchao.py to version guard floatx Created new test to verify version guard of floatx support * Adding torchao version guard for floatx usage Summary: TorchAO removing floatx support, added version guard in quantization_config.py Altered tests in test_torchao.py to version guard floatx Created new test to verify version guard of floatx support * Adding torchao version guard for floatx usage Summary: TorchAO removing floatx support, added version guard in quantization_config.py Altered tests in test_torchao.py to version guard floatx Created new test to verify version guard of floatx support * Adding torchao version guard for floatx usage Summary: TorchAO removing floatx support, added version guard in quantization_config.py Altered tests in test_torchao.py to version guard floatx Created new test to verify version guard of floatx support --------- Co-authored-by: Sayak Paul * Bugfix for dreambooth flux2 img2img2 (#12825) Co-authored-by: Sayak Paul Co-authored-by: Linoy Tsaban <57615435+linoytsaban@users.noreply.github.com> * [Modular] qwen refactor (#12872) * 3 files * add conditoinal pipeline * refactor qwen modular * add layered * up up * u p * add to import * more refacotr, make layer work * clean up a bit git add src * more * style * style * [modular] Tests for custom blocks in modular diffusers (#12557) * start custom block testing. * simplify modular workflow ci. * up * style. * up * up * up * up * up * up * Apply suggestions from code review * up * [chore] remove controlnet implementations outside controlnet module. (#12152) * remove controlnet implementations outside controlnet module. * fix * fix * fix * [core] Handle progress bar and logging in distributed environments (#12806) * disable progressbar in distributed. * up * up * up * up * up * up * Improve docstrings and type hints in scheduling_consistency_models.py (#12931) docs: improve docstring scheduling_consistency_models.py * [Feature] MultiControlNet support for SD3Impainting (#11251) * update * update * addressed PR comments * update * Apply suggestions from code review --------- Co-authored-by: YiYi Xu * Laplace Scheduler for DDPM (#11320) * Add Laplace scheduler that samples more around mid-range noise levels (around log SNR=0), increasing performance (lower FID) with faster convergence speed, and robust to resolution and objective. Reference: https://arxiv.org/pdf/2407.03297. * Fix copies. * Apply style fixes --------- Co-authored-by: YiYi Xu Co-authored-by: github-actions[bot] * Store vae.config.scaling_factor to prevent missing attr reference (sdxl advanced dreambooth training script) (#12346) Store vae.config.scaling_factor to prevent missing attr reference In sdxl advanced dreambooth training script vae.config.scaling_factor becomes inaccessible after: del vae when: --cache_latents, and no --validation_prompt Co-authored-by: Teriks Co-authored-by: Linoy Tsaban <57615435+linoytsaban@users.noreply.github.com> Co-authored-by: YiYi Xu * Add thread-safe wrappers for components in pipeline (examples/server-async/utils/requestscopedpipeline.py) (#12515) * Basic implementation of request scheduling * Basic editing in SD and Flux Pipelines * Small Fix * Fix * Update for more pipelines * Add examples/server-async * Add examples/server-async * Updated RequestScopedPipeline to handle a single tokenizer lock to avoid race conditions * Fix * Fix _TokenizerLockWrapper * Fix _TokenizerLockWrapper * Delete _TokenizerLockWrapper * Fix tokenizer * Update examples/server-async * Fix server-async * Optimizations in examples/server-async * We keep the implementation simple in examples/server-async * Update examples/server-async/README.md * Update examples/server-async/README.md for changes to tokenizer locks and backward-compatible retrieve_timesteps * The changes to the diffusers core have been undone and all logic is being moved to exmaples/server-async * Update examples/server-async/utils/* * Fix BaseAsyncScheduler * Rollback in the core of the diffusers * Update examples/server-async/README.md * Complete rollback of diffusers core files * Simple implementation of an asynchronous server compatible with SD3-3.5 and Flux Pipelines * Update examples/server-async/README.md * Fixed import errors in 'examples/server-async/serverasync.py' * Flux Pipeline Discard * Update examples/server-async/README.md * Apply style fixes * Add thread-safe wrappers for components in pipeline Refactor requestscopedpipeline.py to add thread-safe wrappers for tokenizer, VAE, and image processor. Introduce locking mechanisms to ensure thread safety during concurrent access. * Add wrappers.py * Apply style fixes --------- Co-authored-by: Sayak Paul Co-authored-by: github-actions[bot] * [Research] Latent Perceptual Loss (LPL) for Stable Diffusion XL (#11573) * initial * added readme * fix formatting * added logging * formatting * use config * debug * better * handle SNR * floats have no item() * remove debug * formatting * add paper link * acknowledge reference source * rename script --------- Co-authored-by: Sayak Paul * Change timestep device to cpu for xla (#11501) * Change timestep device to cpu for xla * Add all pipelines * ruff format * Apply style fixes --------- Co-authored-by: YiYi Xu Co-authored-by: github-actions[bot] * [LoRA] add lora_alpha to sana README (#11780) add lora alpha to readme * Fix wrong param types, docs, and handles noise=None in scale_noise of FlowMatching schedulers (#11669) * Bug: Fix wrong params, docs, and handles noise=None * make noise a required arg --------- Co-authored-by: YiYi Xu * [docs] Remote inference (#12372) * init * fix * Align HunyuanVideoConditionEmbedding with CombinedTimestepGuidanceTextProjEmbeddings (#12316) conditioning additions inline with CombinedTimestepGuidanceTextProjEmbeddings Co-authored-by: Samu Tamminen Co-authored-by: YiYi Xu * [Fix] syntax in QwenImageEditPlusPipeline (#12371) * Fixes syntax for consistency among pipelines * Update test_qwenimage_edit_plus.py * Fix ftfy name error in Wan pipeline (#12314) Signed-off-by: Daniel Socek Co-authored-by: YiYi Xu * [modular] error early in `enable_auto_cpu_offload` (#12578) error early in auto_cpu_offload * [ChronoEdit] support multiple loras (#12679) Co-authored-by: wjay Co-authored-by: Dhruv Nair * fix how `is_fsdp` is determined (#12960) up * [LoRA] add LoRA support to LTX-2 (#12933) * up * fixes * tests * docs. * fix * change loading info. * up * up * Fix: typo in autoencoder_dc.py (#12687) Fix typo in autoencoder_dc.py Fixing typo in `get_block` function's parameter name: "qkv_mutliscales" -> "qkv_multiscales" Co-authored-by: YiYi Xu * [Modular] better docstring (#12932) add output to auto blocks + core denoising block for better doc string * [docs] polish caching docs. (#12684) * polish caching docs. * Update docs/source/en/optimization/cache.md Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Update docs/source/en/optimization/cache.md Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * up --------- Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Fix typos (#12705) * Fix link to diffedit implementation reference (#12708) * Fix QwenImage txt_seq_lens handling (#12702) * Fix QwenImage txt_seq_lens handling * formatting * formatting * remove txt_seq_lens and use bool mask * use compute_text_seq_len_from_mask * add seq_lens to dispatch_attention_fn * use joint_seq_lens * remove unused index_block * WIP: Remove seq_lens parameter and use mask-based approach - Remove seq_lens parameter from dispatch_attention_fn - Update varlen backends to extract seqlens from masks - Update QwenImage to pass 2D joint_attention_mask - Fix native backend to handle 2D boolean masks - Fix sage_varlen seqlens_q to match seqlens_k for self-attention Note: sage_varlen still producing black images, needs further investigation * fix formatting * undo sage changes * xformers support * hub fix * fix torch compile issues * fix tests * use _prepare_attn_mask_native * proper deprecation notice * add deprecate to txt_seq_lens * Update src/diffusers/models/transformers/transformer_qwenimage.py Co-authored-by: YiYi Xu * Update src/diffusers/models/transformers/transformer_qwenimage.py Co-authored-by: YiYi Xu * Only create the mask if there's actual padding * fix order of docstrings * Adds performance benchmarks and optimization details for QwenImage Enhances documentation with comprehensive performance insights for QwenImage pipeline: * rope_text_seq_len = text_seq_len * rename to max_txt_seq_len * removed deprecated args * undo unrelated change * Updates QwenImage performance documentation Removes detailed attention backend benchmarks and simplifies torch.compile performance description Focuses on key performance improvement with torch.compile, highlighting the specific speedup from 4.70s to 1.93s on an A100 GPU Streamlines the documentation to provide more concise and actionable performance insights * Updates deprecation warnings for txt_seq_lens parameter Extends deprecation timeline for txt_seq_lens from version 0.37.0 to 0.39.0 across multiple Qwen image-related models Adds a new unit test to verify the deprecation warning behavior for the txt_seq_lens parameter * fix compile * formatting * fix compile tests * rename helper * remove duplicate * smaller values * removed * use torch.cond for torch compile * Construct joint attention mask once * test different backends * construct joint attention mask once to avoid reconstructing in every block * Update src/diffusers/models/attention_dispatch.py Co-authored-by: YiYi Xu * formatting * raising an error from the EditPlus pipeline when batch_size > 1 --------- Co-authored-by: Sayak Paul Co-authored-by: YiYi Xu Co-authored-by: cdutr * Bugfix for flux2 img2img2 prediction (#12855) * Bugfix for dreambooth flux2 img2img2 * Bugfix for dreambooth flux2 img2img2 * Bugfix for dreambooth flux2 img2img2 * Bugfix for dreambooth flux2 img2img2 * Bugfix for dreambooth flux2 img2img2 * Bugfix for dreambooth flux2 img2img2 Co-authored-by: tcaimm <93749364+tcaimm@users.noreply.github.com> --------- Co-authored-by: tcaimm <93749364+tcaimm@users.noreply.github.com> * Add Flag to `PeftLoraLoaderMixinTests` to Enable/Disable Text Encoder LoRA Tests (#12962) * Improve incorrect LoRA format error message * Add flag in PeftLoraLoaderMixinTests to disable text encoder LoRA tests * Apply changes to LTX2LoraTests * Further improve incorrect LoRA format error msg following review --------- Co-authored-by: Sayak Paul * Add Unified Sequence Parallel attention (#12693) * initial scheme of unified-sp * initial all_to_all_double * bug fixes, added cmnts * unified attention prototype done * remove raising value error in contextParallelConfig to enable unified attention * bug fix * feat: Adds Test for Unified SP Attention and Fixes a bug in Template Ring Attention * bug fix, lse calculation, testing bug fixes, lse calculation - switched to _all_to_all_single helper in _all_to_all_dim_exchange due contiguity issues bug fix bug fix bug fix * addressing comments * sequence parallelsim bug fixes * code format fixes * Apply style fixes * code formatting fix * added unified attention docs and removed test file * Apply style fixes * tip for unified attention in docs at distributed_inference.md Co-authored-by: Sayak Paul * Update distributed_inference.md, adding benchmarks Co-authored-by: Sayak Paul * Update docs/source/en/training/distributed_inference.md Co-authored-by: Sayak Paul * function name fix * fixed benchmark in docs --------- Co-authored-by: KarthikSundar2002 Co-authored-by: github-actions[bot] Co-authored-by: Sayak Paul * [Modular] Changes for using WAN I2V (#12959) * initial * add kayers * Z rz rz rz rz rz rz r cogview (#12973) * init * add * add 1 * Update __init__.py * rename * 2 * update * init with encoder * merge2pipeline * Update pipeline_glm_image.py * remove sop * remove useless func * Update pipeline_glm_image.py * up (cherry picked from commit cfe19a31b9bc14c090c7259c09f3532dfafcd059) * review for work only * change place * Update pipeline_glm_image.py * update * Update transformer_glm_image.py * 1 * no negative_prompt for GLM-Image * remove CogView4LoraLoaderMixin * refactor attention processor. * update * fix * use staticmethod * update * up * up * update * Update glm_image.md * 1 * Update pipeline_glm_image.py * Update transformer_glm_image.py * using new transformers impl * support * resolution change * fix-copies * Update src/diffusers/pipelines/glm_image/pipeline_glm_image.py Co-authored-by: YiYi Xu * Update pipeline_glm_image.py * use cogview4 * Update pipeline_glm_image.py * Update pipeline_glm_image.py * revert * update * batch support * update * version guard glm image pipeline * validate prompt_embeds and prior_token_ids * try docs. * 4 * up * up * skip properly * fix tests * up * up --------- Co-authored-by: zRzRzRzRzRzRzR <2448370773@qq.com> Co-authored-by: yiyixuxu * Update distributed_inference.md to reposition sections (#12971) * [chore] make transformers version check stricter for glm image. (#12974) * make transformers version check stricter for glm image. * public checkpoint. * Remove 8bit device restriction (#12972) * allow to * update version * fix version again * again * Update src/diffusers/pipelines/pipeline_utils.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * style * xfail * add pr --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Sayak Paul * `disable_mmap` in pipeline `from_pretrained` (#12854) * update * `disable_mmap` in `from_pretrained` --------- Co-authored-by: DN6 * [Modular] mellon utils (#12978) * up * style --------- Co-authored-by: yiyi@huggingface.co * LongCat Image pipeline: Allow offloading/quantization of text_encoder component (#12963) * Don't attempt to move the text_encoder. Just move the generated_ids. * The inputs to the text_encoder should be on its device * Add `ChromaInpaintPipeline` (#12848) * Add `ChromaInpaintPipeline` * Set `attention_mask` to `dtype=torch.bool` for `ChromaInpaintPipeline`. * Revert `.gitignore`. * fix Qwen-Image series context parallel (#12970) * fix qwen-image cp * relax attn_mask limit for cp * CP plan compatible with zero_cond_t * move modulate_index plan to top level * Flux2 klein (#12982) * flux2-klein * Apply suggestions from code review Co-authored-by: Sayak Paul * Klein tests (#2) * tests * up * tests * up * support step-distilled * Apply suggestions from code review Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> * doc string etc * style * more * copies * klein lora training scripts (#3) * initial commit * initial commit * remove remote text encoder * initial commit * initial commit * initial commit * revert * img2img fix * text encoder + tokenizer * text encoder + tokenizer * update readme * guidance * guidance * guidance * test * test * revert changes not needed for the non klein model * Update examples/dreambooth/train_dreambooth_lora_flux2_klein.py Co-authored-by: Sayak Paul * fix guidance * fix validation * fix validation * fix validation * fix path * space --------- Co-authored-by: Sayak Paul * style * Update src/diffusers/pipelines/flux2/pipeline_flux2_klein.py * Apply style fixes * auto pipeline --------- Co-authored-by: Sayak Paul Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> Co-authored-by: Linoy Tsaban <57615435+linoytsaban@users.noreply.github.com> Co-authored-by: github-actions[bot] * [modular] fix a bug in mellon param & improve docstrings (#12980) * update mellonparams docstring to incude the acutal param definition render in mellon * style --------- Co-authored-by: yiyi@huggingface.co * add klein docs. (#12984) * LTX 2 Single File Support (#12983) * LTX 2 transformer single file support * LTX 2 video VAE single file support * LTX 2 audio VAE single file support * Make it easier to distinguish LTX 1 and 2 models * [core] gracefully error out when attn-backend x cp combo isn't supported. (#12832) * gracefully error out when attn-backend x cp combo isn't supported. * Revert "gracefully error out when attn-backend x cp combo isn't supported." This reverts commit c8abb5d7c01ca6a7c0bf82c27c91a326155a5e43. * gracefully error out when attn-backend x cp combo isn't supported. * up * address PR feedback. * up * Update src/diffusers/models/modeling_utils.py Co-authored-by: Dhruv Nair * dot. --------- Co-authored-by: Dhruv Nair * Improve docstrings and type hints in scheduling_cosine_dpmsolver_multistep.py (#12936) * docs: improve docstring scheduling_cosine_dpmsolver_multistep.py * Update src/diffusers/schedulers/scheduling_cosine_dpmsolver_multistep.py Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Update src/diffusers/schedulers/scheduling_cosine_dpmsolver_multistep.py Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * fix --------- Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * [Docs] Replace root CONTRIBUTING.md with symlink to source docs (#12986) Chore: Replace CONTRIBUTING.md with a symlink to documentation * make style && make quality * Revert "make style && make quality" This reverts commit 76f51a5e927b369b5f708583e8f18b49c350f52a. * [chore] make style to push new changes. (#12998) make style to push new changes. * Fibo edit pipeline (#12930) * Feature: Add BriaFiboEditPipeline to diffusers * Introduced BriaFiboEditPipeline class with necessary backend requirements. * Updated import structures in relevant modules to include BriaFiboEditPipeline. * Ensured compatibility with existing pipelines and type checking. * Feature: Introduce Bria Fibo Edit Pipeline * Added BriaFiboEditPipeline class for structured JSON-native image editing. * Created documentation for the new pipeline in bria_fibo_edit.md. * Updated import structures to include the new pipeline and its components. * Added unit tests for the BriaFiboEditPipeline to ensure functionality and correctness. * Enhancement: Update Bria Fibo Edit Pipeline and Documentation * Refined the Bria Fibo Edit model description for clarity and detail. * Added usage instructions for model authentication and login. * Implemented mask handling functions in the BriaFiboEditPipeline for improved image editing capabilities. * Updated unit tests to cover new mask functionalities and ensure input validation. * Adjusted example code in documentation to reflect changes in the pipeline's usage. * Update Bria Fibo Edit documentation with corrected Hugging Face page link * add dreambooth training script * style and quality * Delete temp.py * Enhancement: Improve JSON caption validation in DreamBoothDataset * Updated the clean_json_caption function to handle both string and dictionary inputs for captions. * Added error handling to raise a ValueError for invalid caption types, ensuring better input validation. * Add datasets dependency to requirements_fibo_edit.txt * Add bria_fibo_edit to docs table of contents * Fix dummy objects ordering * Fix BriaFiboEditPipeline to use passed generator parameter The pipeline was ignoring the generator parameter and only using the seed parameter. This caused non-deterministic outputs in tests that pass a seeded generator. * Remove fibo_edit training script and related files --------- Co-authored-by: kfirbria * Fix variable name in docstring for PeftAdapterMixin.set_adapters (#13003) Co-authored-by: Sayak Paul * Improve docstrings and type hints in scheduling_ddim_cogvideox.py (#12992) docs: improve docstring scheduling_ddim_cogvideox.py * [scheduler] Support custom sigmas in UniPCMultistepScheduler (#12109) * update * fix tests * Apply suggestions from code review * Revert default flow sigmas change so that tests relying on UniPC multistep still pass * Remove custom timesteps for UniPC multistep set_timesteps --------- Co-authored-by: YiYi Xu Co-authored-by: Daniel Gu Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> * feat: accelerate longcat-image with regional compile (#13019) * Improve docstrings and type hints in scheduling_ddim_flax.py (#13010) * docs: improve docstring scheduling_ddim_flax.py * docs: improve docstring scheduling_ddim_flax.py * docs: improve docstring scheduling_ddim_flax.py * Improve docstrings and type hints in scheduling_ddim_inverse.py (#13020) docs: improve docstring scheduling_ddim_inverse.py * fix Dockerfiles for cuda and xformers. (#13022) * Resnet only use contiguous in training mode. (#12977) * fix contiguous Signed-off-by: jiqing-feng * update tol Signed-off-by: jiqing-feng * bigger tol Signed-off-by: jiqing-feng * fix tests Signed-off-by: jiqing-feng * update tol Signed-off-by: jiqing-feng --------- Signed-off-by: jiqing-feng Co-authored-by: Sayak Paul * feat: add qkv projection fuse for longcat transformers (#13021) feat: add qkv fuse for longcat transformers Co-authored-by: Sayak Paul * Improve docstrings and type hints in scheduling_ddim_parallel.py (#13023) * docs: improve docstring scheduling_ddim_parallel.py * docs: improve docstring scheduling_ddim_parallel.py * Update src/diffusers/schedulers/scheduling_ddim_parallel.py Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Update src/diffusers/schedulers/scheduling_ddim_parallel.py Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Update src/diffusers/schedulers/scheduling_ddim_parallel.py Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Update src/diffusers/schedulers/scheduling_ddim_parallel.py Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * fix style --------- Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Improve docstrings and type hints in scheduling_ddpm_flax.py (#13024) docs: improve docstring scheduling_ddpm_flax.py * Improve docstrings and type hints in scheduling_ddpm_parallel.py (#13027) * docs: improve docstring scheduling_ddpm_parallel.py * Update scheduling_ddpm_parallel.py Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> --------- Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Remove `*pooled_*` mentions from Chroma inpaint (#13026) Remove `*pooled_*` mentions from Chroma as it has just one TE. * Flag Flax schedulers as deprecated (#13031) flag flax schedulers as deprecated * [modular] add auto_docstring & more doc related refactors (#12958) * up * up up * update outputs * style * add modular_auto_docstring! * more auto docstring * style * up up up * more more * up * address feedbacks * add TODO in the description for empty docstring * refactor based on dhruv's feedback: remove the class method * add template method * up * up up up * apply auto docstring * make style * rmove space in make docstring * Apply suggestions from code review * revert change in z * fix * Apply style fixes * include auto-docstring check in the modular ci. (#13004) * Run ruff format after auto docstring generation * up * upup * upup * style --------- Co-authored-by: github-actions[bot] Co-authored-by: Sayak Paul * Upgrade GitHub Actions to latest versions (#12866) * Upgrade GitHub Actions to latest versions Signed-off-by: Salman Muin Kayser Chishti <13schishti@gmail.com> * fix: Correct GitHub Actions upgrade (fix branch refs and version formats) * fix: Correct GitHub Actions upgrade (fix branch refs and version formats) * fix: Correct GitHub Actions upgrade (fix branch refs and version formats) --------- Signed-off-by: Salman Muin Kayser Chishti <13schishti@gmail.com> * [From Single File] support `from_single_file` method for `WanAnimateTransformer3DModel` (#12691) * Add `WanAnimateTransformer3DModel` to `SINGLE_FILE_LOADABLE_CLASSES` * Fixed dtype mismatch when loading a single file * Fixed a bug that results in white noise for generation * Update dtype check for time embedder - caused white noise output * Improve code readability * Optimize dtype handling Removed unnecessary dtype conversions for timestep and weight. * Apply style fixes * Refactor time embedding dtype handling Adjust time embedding type conversion for compatibility. * Apply style fixes * Modify comment for WanTimeTextImageEmbedding class --------- Co-authored-by: Sam Edwards Co-authored-by: github-actions[bot] * Fix: Cosmos2.5 Video2World frame extraction and add default negative prompt (#13018) * fix: Extract last frames for conditioning in Cosmos Video2World * Added default negative prompt * Apply style fixes * Added default negative prompt in cosmos2 text2image pipeline --------- Co-authored-by: YiYi Xu Co-authored-by: github-actions[bot] * [GLM-Image] Add batch support for GlmImagePipeline (#13007) * init Signed-off-by: JaredforReal * change from right padding to left padding Signed-off-by: JaredforReal * try i2i batch Signed-off-by: JaredforReal * fix: revert i2i prior_token_image_ids to original 1D tensor format * refactor KVCache for per prompt batching Signed-off-by: JaredforReal * fix KVCache Signed-off-by: JaredforReal * fix shape error Signed-off-by: JaredforReal * refactor pipeline Signed-off-by: JaredforReal * fix for left padding Signed-off-by: JaredforReal * insert seed to AR model Signed-off-by: JaredforReal * delete generator, use torch manual_seed Signed-off-by: JaredforReal * add batch processing unit tests for GlmImagePipeline Signed-off-by: JaredforReal * simplify normalize images method Signed-off-by: JaredforReal * fix grids_per_sample Signed-off-by: JaredforReal * fix t2i Signed-off-by: JaredforReal * delete comments, simplify condition statement Signed-off-by: JaredforReal * chage generate_prior_tokens outputs Signed-off-by: JaredforReal * simplify if logic Signed-off-by: JaredforReal * support user provided prior_token_ids directly Signed-off-by: JaredforReal * remove blank lines Signed-off-by: JaredforReal * align with transformers Signed-off-by: JaredforReal * Apply style fixes --------- Signed-off-by: JaredforReal Co-authored-by: github-actions[bot] * [Qwen] avoid creating attention masks when there is no padding (#12987) * avoid creating attention masks when there is no padding * make fix-copies * torch compile tests * set all ones mask to none * fix positional encoding from becoming > 4096 * fix from review * slice freqs_cis to match the input sequence length * keep only attenton masking change --------- Co-authored-by: Sayak Paul * [modular]support klein (#13002) * support klein * style * copies * Apply suggestions from code review Co-authored-by: Sayak Paul Co-authored-by: Álvaro Somoza * Update src/diffusers/modular_pipelines/flux2/encoders.py * a few fix: unpack latents before decoder etc * style * remove guidannce to its own block * style * flux2-dev work in modular setting * up * up up * add tests --------- Co-authored-by: yiyi@huggingface.co Co-authored-by: Sayak Paul Co-authored-by: Álvaro Somoza * [QwenImage] fix prompt isolation tests (#13042) * up * up * up * fix * fast tok update (#13036) * v5 tok update * ruff * keep pre v5 slow code path * Apply style fixes --------- Co-authored-by: Sayak Paul Co-authored-by: github-actions[bot] * change to CUDA 12.9. (#13045) * change to CUDA 12.9. * up * change runtime base * FROM * remove torchao autoquant from diffusers docs (#13048) Summary: Context: https://github.com/pytorch/ao/issues/3739 Test Plan: CI, since this does not change any Python code * docs: improve docstring scheduling_dpm_cogvideox.py (#13044) * Fix Wan/WanI2V patchification (#13038) * Fix Wan/WanI2V patchification * Apply style fixes * Apply suggestions from code review I agree with you for the idea of using `patch_size` instead. Thanks!😊 Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> * Fix logger warning --------- Co-authored-by: github-actions[bot] Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> * LTX2 distilled checkpoint support (#12934) * add constants for distill sigmas values and allow ltx pipeline to pass in sigmas * add time conditioning conversion and token packing for latents * make style & quality * remove prenorm * add sigma param to ltx2 i2v * fix copies and add pack latents to i2v * Apply suggestions from code review Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> * Infer latent dims if latents/audio_latents is supplied * add note for predefined sigmas * run make style and quality * revert distill timesteps & set original_state_dict_repo_idd to default None * add latent normalize * add create noised state, delete last sigmas * remove normalize step in latent upsample pipeline and move it to ltx2 pipeline * add create noise latent to i2v pipeline * fix copies * parse none value in weight conversion script * explicit shape handling * Apply suggestions from code review Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> * make style * add two stage inference tests * add ltx2 documentation * update i2v expected_audio_slice * Apply suggestions from code review Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> * Apply suggestion from @dg845 Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> * Update ltx2.md to remove one-stage example Removed one-stage generation example code and added comments for noise scale in two-stage generation. --------- Co-authored-by: Sayak Paul Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> Co-authored-by: Daniel Gu * [wan] fix layerwise upcasting tests on CPU (#13039) up * [ci] uniform run times and wheels for pytorch cuda. (#13047) * uniform run times and wheels for pytorch cuda. * 12.9 * change to 24.04. * change to 24.04. * docs: fix grammar in fp16_safetensors CLI warning (#13040) * docs: fix grammar in fp16_safetensors CLI warning * Apply style fixes --------- Co-authored-by: github-actions[bot] * [wan] fix wan 2.2 when either of the transformers isn't present. (#13055) fix wan 2.2 when either of the transformers isn't present. * [bug fix] GLM-Image fit new `get_image_features` API (#13052) change get_image_features API Signed-off-by: JaredforReal Co-authored-by: YiYi Xu * Fix aiter availability check (#13059) Update import_utils.py * [Modular]add a real quick start guide (#13029) * add a real quick start guide * Update docs/source/en/modular_diffusers/quickstart.md * update a bit more * fix * Apply suggestions from code review Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Update docs/source/en/modular_diffusers/quickstart.md Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Update docs/source/en/modular_diffusers/quickstart.md Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * update more * Apply suggestions from code review Co-authored-by: Sayak Paul * address more feedbacks: move components amnager earlier, explain blocks vs sub-blocks etc * more * remove the link to mellon guide, not exist in this PR yet --------- Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> Co-authored-by: Sayak Paul * feat: support Ulysses Anything Attention (#12996) * feat: support Ulysses Anything Attention * feat: support Ulysses Anything Attention * feat: support Ulysses Anything Attention * feat: support Ulysses Anything Attention * fix UAA broken while using joint attn * update * post check * add docs * add docs * remove lru cache * move codes * update * Refactor Model Tests (#12822) * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update --------- Co-authored-by: Sayak Paul * [Flux2] Fix LoRA loading for Flux2 Klein by adaptively enumerating transformer blocks (#13030) * Resolve Flux2 Klein 4B/9B LoRA loading errors * Apply style fixes --------- Co-authored-by: github-actions[bot] Co-authored-by: Sayak Paul * [Modular] loader related (#13025) * tag loader_id from Automodel * style * load_components by default only load components that are not already loaded * by default, skip loading the componeneets does not have the repo id * [Modular] mellon doc etc (#13051) * add metadata field to input/output param * refactor mellonparam: move the template outside, add metaclass, define some generic template for custom node * add from_custom_block * style * up up fix * add mellon guide * add to toctree * style * add mellon_types * style * mellon_type -> inpnt_types + output_types * update doc * add quant info to components manager * fix more * up up * fix components manager * update custom block guide * update * style * add a warn for mellon and add new guides to overview * Apply suggestions from code review Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Update docs/source/en/modular_diffusers/mellon.md Co-authored-by: Ste…

Sunbelt Computer Software

PL/B Language Development and Support