Activity

1. yiyixuxu

   added

   diffusers-mvp

   on Aug 18, 2026
2. yiyixuxu

   assigned

   yiyixuxu

   ,

   DN6

   and

   sayakpaul

   on Aug 18, 2026
3. yiyixuxu

   pinned this issue on Aug 18, 2026
4. lucasruan1618 commented on Aug 19, 2026

   

   lucasruan1618

   on Aug 19, 2026

   Contributor

   More actions

   I’d like to propose two ideas that could strengthen Diffusers as a foundation for the broader generative-model ecosystem: one focused on making models easier to run efficiently across different hardware, and another focused on making their behavior easier to verify across different runtimes.

   1. Hardware-aware execution profiles

   Diffusers has made diffusion models portable across an enormous range of architectures, but running those models is still surprisingly hardware-specific.

   Today, users often need to understand and manually combine quantization, offloading, device placement, attention backends, compilation, VAE memory strategies, and precision choices just to answer a simple question:

   “What is the best way to run this model on my hardware?”

   The problem becomes especially visible with modern video and audio models, where an execution strategy can determine whether a model fits at all.

   I propose making hardware-aware execution profiles a first-class concept in Diffusers.

   A user should be able to describe an execution goal — for example, fit within available VRAM, minimize latency under a memory budget, or reduce memory without quality-changing approximations — and have Diffusers provide a validated execution profile for that model and hardware, with the relevant trade-offs made explicit.

   This would turn Diffusers' growing collection of optimization capabilities into something much more accessible: not just model portability, but execution portability.

   It would make large generative models accessible to a much broader hardware range, reduce fragile model-specific optimization recipes scattered across examples, and give model authors a common way to communicate how their models can practically be run.

   ---

   2. Diffusers as an executable reference specification

   As diffusion models increasingly run through visual workflow tools, optimized serving engines, distributed runtimes, and hardware-specific implementations, sharing the same weights no longer guarantees sharing the same model behavior.

   This becomes particularly important as runtimes introduce increasingly aggressive performance optimizations such as sequence/tensor parallelism, specialized attention kernels, quantization, caching, and alternative execution backends.

   The ecosystem is already running into this problem:

   • SGLang #19164 explicitly calls for correctness testing to ensure SP/TP and specialized kernels do not degrade diffusion quality.
   • vLLM-Omni #3256 reports a substantial Qwen-Image-2512 quality regression relative to its Diffusers reference.
   • vLLM-Omni #4166 shows an LTX-2.3 full-pipeline similarity test failing against Diffusers.

   These projects are effectively building their own conformance infrastructure today.

   I propose making cross-runtime conformance a first-class role for Diffusers.

   For supported model families, Diffusers could provide a portable, machine-verifiable definition of reference behavior. Optimized runtimes could validate that performance improvements preserve the intended model semantics rather than relying on visual inspection or ad-hoc model-specific tests.

   This would make Diffusers more than one implementation among many: it could become the executable reference specification for open generative models.

   Model authors would gain a canonical behavior to point to, runtime developers a shared correctness target, and production users a much safer way to adopt faster backends and aggressive optimizations.
5. PrakshaaleJain commented on Aug 20, 2026

   

   PrakshaaleJain

   on Aug 20, 2026

   Last edited by PrakshaaleJain

   Contributor

   More actions

   I would like to propose an idea for a pain-point I experienced firsthand when I was using diffusers library a while back when I was working with diffusion models. This pain-point is about the lack of standardization in the prompt weighting techniques used for image generation.

   Idea: Universal Prompt Parser and Syntax Translation Engine

   What it is:
   Diffusers is the undeniable backbone of open-source image generation. However, it currently outsources complex text parsing to third-party libraries like compel.

   Outsourcing this creates a dangerous leaky abstraction as models scale to massive multi-encoder architectures like Stable Diffusion XL. Relying on external community libraries severely fragments the developer experience and creates a massive risk of text-processing breaking entirely when cutting-edge models are released.

   Furthermore, outsourcing prevents standardization in the community. Users coming from Automatic1111 or Midjourney all have different muscle memory for weighting tokens in the prompts (e.g., (word:1.5) in Automatic1111 and word::1.5 in Midjourney).

   I propose building a "Universal Prompt Parser" directly into Diffusers. Rather than enforcing one specific syntax, this native module would act as a translation layer. Users could pass a syntax="a1111" or syntax="midjourney" argument, and Diffusers would seamlessly parse their preferred dialect, handle long-prompt chunking, and calculate the delicate embedding multipliers natively.

   How it benefits Diffusers:

   1. The Ultimate Ecosystem Bridge: By supporting multiple syntax dialects out-of-the-box, Diffusers becomes the universal translation layer. It eliminates the steep learning curve for creators migrating from visual UI tools to Python, completely respecting their existing muscle memory.
   2. Solves the Multi-Encoder Crisis: External libraries are struggling to support the incredibly diverse array of modern text encoders. A native engine guarantees that prompt weighting works safely across CLIP, T5, and future architectures, proactively preventing out-of-distribution math artifacts.
   3. Seamless Developer Experience: It permanently patches a major ecosystem leak. Developers get the exact creative control of visual UIs directly in Python, without ever leaving the Hugging Face ecosystem or juggling fragile, hidden prompt embedding math.
6. hlky commented on Aug 20, 2026

   

   hlky

   on Aug 20, 2026

   Contributor

   More actions

   1. Diffusers as reference spec, single-file implementations, documented inputs and ranges/limits, etc.
   2. Make pipelines interactive/resumable, more than callback hooks, and session based e.g. session stores prompt embeds, latents at step, etc.
   3. Ablation/research/education, experiment/modify/inspect without forking
   4. Extend configs with capabilities, supports cfg, supports negative prompt, etc.
   5. Standardized provenance
   6. Batch/dataset production workflows
   7. Resource aware automatic selection of tiling, offloading, quant, attention backend, etc. with tradeoff of quality, latency, vram etc.
7. phoeenniixx commented on Aug 22, 2026

   

   phoeenniixx

   on Aug 22, 2026

   More actions

   Idea: run observability and divergence localization for diffusion pipelines

   Between __call__ and the final output, a pipeline is a black box. The step callback exists and Modular's PipelineState exposes intermediates, but there is no tooling on top of them. Debugging is still print statements and breakpoints, and for these bugs it is not clear where to even put them.

   Silent numerical bugs are the worst case: nothing errors, the output just degrades. Eg #12075, #12294, or a LoRA loading as a no-op. Parity reports like #12216 stay open for the same reason: nobody can point to the step where behavior departs.

   The idea: make every pipeline run recordable, and any two recordings comparable. A recording captures inputs, config and intermediate states. A comparison localizes the first point of divergence: across diffusers versions, across settings, dtypes and optimizations, or against an external implementation.

   Think TensorBoard for inference runs, except the comparison is automated: it reports the first step where two runs depart instead of leaving you to eyeball curves.

   This can be helpful:

   • "Attach your recording" on bug reports, like a stack trace. Subjective quality reports become localized divergences, less triage for maintainers.
   • Catches the silent-failure class automatically.
   • Complements @lucasruan1618's reference-spec proposal above: conformance tells a runtime that it diverged, this tells it where. The vLLM-Omni regressions cited there end in exactly that question.

   Precedent exists in transformers already: model_addition_debugger_context traces a forward so model adders can compare against a reference. Diffusion has no equivalent, and its divergences mostly live outside a single forward, in the scheduler math, guidance, prompt encoding and the denoising loop. The one prior tool, diffusers-interpret, is abandoned.

   🚀React with 🚀2Reacted by Lucas Ruan and Devam0311
8. iwr-redmond commented on Aug 23, 2026

   

   iwr-redmond

   on Aug 23, 2026

   Last edited by iwr-redmond

   More actions

   This suggestion is process rather than feature-oriented and relates to the release schedule. The decision processes regarding releases are currently unclear, which leads to issues such as:

   • Requests for more information about the timetable
   • Items remaining on the carried-over Release Plan for over a year (e.g. Flux.1 cannot load standard transformer in nf4 #9996)
   • Downstream applications choosing Git commits over official releases
   • Potentially useful bugfixes going unreviewed for months (e.g. Fix ControlNet img2img and inpaint control image validation #14069) as there is no pre-release sprint - although there has been considerable internal work on tests (e.g. [tests] Add Config to Exclude Modules from Leaf-Level Group Offloading #14564, [tests] skip parallelism tests for some models. #14505, [CI] Allow single file tests to run as fast tests  #14324, etc.)

   You may wish to consider transitioning to a more helpful release strategy based on ideas developed in other projects.

   Strategy	Example	Potential Benefits
   Rolling releases	Gradio	Regular minor releases Breaking changes trigger version bump Easier for downstream integrators
   Versioned releases + rolling prereleases	Llama.cpp	Balances stability and frequency Regular builds for early adopters Developers can opt-in by specifying a particular prerelease version
   Pre-defined schedule	Ubuntu	Predictable schedule for new versions Shapes third-party developer behavior over time Option for dedicated bugfixing period prior to final releases

   The current "muddle-through" strategy is closest to a rolling release schedule, with limitations include breaking changes occurring in every new version and no public bugfixing strategy. If there were to be a change, the easiest would probably be to release more frequently, probably by adopting Gradio's cadence of roughly 2-3 releases per month. However, I reckon that the best strategy would be to adopt a fixed quarterly release schedule with a period of about two weeks between the release candidate and final release dedicated to bug-fixing. This would allow for a focus on quality and encourage PR authors to submit major new features early in the release cycle. You could optionally provide regular pre-releases for eager early adopters while remaining focused on the quarterly schedule.
9. iwr-redmond commented on Aug 23, 2026

   

   iwr-redmond

   on Aug 23, 2026

   More actions

   I will also draw attention to my previous suggestion about PR reviews in #13452, which is happily under 300 words as-is 😉