A1111 <> Diffusers Scheduler mapping

A1111 <> Diffusers Mapping

Diffusers samplers have a couple more ways to be customized as can be seen from the inits of the respective samplers.

@stevhliu

@stevhliu could we maybe put this in a doc as discussed a bit during our meeting? Maybe in the overview doc page?

Nice, I'll roll this into a PR I'm working on to update the scheduler APIs!

@patrickvonplaten

@patrickvonplaten: I'm confused: Shouldn't DPM++ SDE be DPMSolverSDEScheduler? And in this issue you just said that "DPM++ 2s a" = DPMSolverSinglestepScheduler.

https://huggingface.co/docs/diffusers/v0.18.2/en/api/schedulers/dpm_sde is another implementation of DPMSolverSDE but it's not the official one (the author contributed DPMSolverSDE himself), so that the default one.
In my opinion, we could deprecate DPMSolverSDE at some point.
RE: DPM++ 2s a yes indeed it's very similar to DPMSolverSinglestepScheduler with init with algorithm_type="sde-dpmsolver++" and use_karras_sigmas=True

@patrickvonplaten

Hi @patrickvonplaten Thanks for the clarification. For DPM++ 2s a - can you please confirm whether the algorithm_type should be sde-dpmsolver++ or dpmsolver++?

Your recommendation in #3953 is dpmsolver++:

DPM++ 2s a => should be the following: DPMSolverSinglestepScheduler.from_config(..., algorithm_type="dpmsolver++")

And your most-recent message here is sde-dpmsolver++:

RE: DPM++ 2s a yes indeed it's very similar to DPMSolverSinglestepScheduler with init with algorithm_type="sde-dpmsolver++" and use_karras_sigmas=True

Thanks!

Hmm, I don't think sde-dpmsolver++ is implemented for DPMSolverSinglestepScheduler

NotImplementedError: sde-dpmsolver++ does is not implemented for <class 'diffusers.schedulers.scheduling_dpmsolver_singlestep.DPMSolverSinglestepScheduler'>

Ah good catch! We need to add this algorithm type. Here: https://github.com/huggingface/diffusers/pull/3344/files was the PR to add it to the Multi-Step scheduler. I think it can be more or less copied 1-to-1 to DPMSolverSinglestepScheduler - would you like to open a PR?

@patrickvonplaten

@patrickvonplaten Thanks! I've opened a PR at #4251

There's a discrepancy that I'm not sure about, I've raised that in the PR.

I tried implementing a1111's kdiffusion samplers in diffusers along with the ability to pass user changable settings from a1111 to kdiffusion.

Do note that I don't have much experience in this field, it's just something I got into for fun the last month or two.

But my findings are:

UniPC is available in a1111, but I was not sure how to translate skip type. A1111 has logSNR, time_uniform and time_quadratic, while diffusers has linspace, leading and trailing?
Samplers with "brownian_noise": True will pass a custom noise generator to kdiffusion that generate brownian noise
From settings you can change the eta of Euler a and other samplers, by default it's 1.0 and it's used here in kdiffusion and I imagine in euler a's case it would go here
You can also change s_churn = 0.0, s_tmin = 0.0, s_tmax = 0.0, s_noise = 1.0, from settings which are passed onto the kdiffusion samplers
A1111 overrides torch.randnlike to produce consistent results in samplers that use them. The random tensors are pushed onto some array and then popped by kdiffusion calling torch.randnlike
- An API for this was requested by AUTOMATIC1111 and solved here [Feature request] Let user provide his own randn data for samplers in sampling.py crowsonkb/k-diffusion#25 but it was never actually used in the webui, apart from the brownian noise find.
The creation of sigmas in a1111 to be passed to a sampler seem a little involved. You can change the "rho", "sigma_min" and "sigma_max" from the ui, and there are some backwards compatible flags

I was going for 1 to 1 kind of for fun, but I think ideally some sacrifices should be made. Like I'm not sure people intentionally use the extra sampler settings, and they're all set to default as far as I know.

My implementation can be found here. I've compared various samplers to a1111 and my implementation and it seems to work, but probably there are details I've missed.
https://github.com/CapsAdmin/diffusers-a1111/

I've refactored and fixed some issues, samplers reproduce the same result in a1111 except for DPM2 a Karras (which is marked as a second order sampler). I also explain a bit better in code what's going on in kdiffusion.py

I can observe that samplers like Heun, DPM2 a, DPM fast, DPM++ SDE Karras produce very similar results but has a subtle variation from a1111.

I was thinking this would be the sigma values being slightly different. For example:

diffusers: [14.6172, 11.7266,  9.3438,  7.3867,  5.7891,  4.5000,  3.4648,  2.6406,
         1.9912,  1.4834,  1.0908,  0.7910,  0.5649,  0.3965,  0.2732,  0.1842,
         0.1214,  0.0779,  0.0485,  0.0292,  0.0000]

a1111: [14.6116, 11.7413,  9.3685,  7.4192,  5.8284,  4.5394,  3.5029,  2.6763,
         2.0228,  1.5112,  1.1147,  0.8110,  0.5812,  0.4096,  0.2834,  0.1921,
         0.1273,  0.0822,  0.0515,  0.0313,  0.0000]

I used the same method from the k-diffusion pipeline in diffusers to get the min max sigma and I'm not sure how to do it otherwise.

I also discovered a potential bug where a1111 will pass eta=1 coming from sampler settings to DPM Fast and DPM Adaptive. The default eta in kdiffusion is 0.

When they are used with eta 0, DPM fast actually produces good results and DPM adaptive finishes a lot faster.

old man, 25 steps, DPM fast, eta 0

old man, 25 steps, DPM fast, eta 1

I think this setting was supposed to be used for ancestral samplers, but accidentally brought to these samplers.

So I guess this is the reason DPM fast is infamous for producing bad results.

@patrickvonplaten

Wow, this is great, thanks @patrickvonplaten , I have been spending lots of nights testing out the correct mapping relationship

@patrickvonplaten

Thank you @patrickvonplaten for writing up this mapping! <3

@stevhliu did it eventually make it into the Diffusers documentation? I just want to make sure whether this mapping above is still the point of truth or if there is another one somewhere I should use...

Side note, I noticed there are a couple of schedulers in diffusers that are not included in the mapping above. It seems the list is not entirely exhaustive, maybe because they do not exist in A1111?

Thank you both! 👐

@alexisrolland

Hi @alexisrolland, the mapping is live on the main version of the docs 🤗

@CodeCorrupt

Hey @CodeCorrupt,

Yes we could add them in a PR I think :-)

@CodeCorrupt

@CodeCorrupt do you mind opening a new issue with this?

New issue was opened #5002 👍

Hi! Wondering if there's plan for "DPM++ 3M SDE" implementation (or if that's already implemented?)

There is no plan to add it yet, feel free to open a PR @JLLLinn :-)

I've refactored and fixed some issues, samplers reproduce the same result in a1111 except for DPM2 a Karras (which is marked as a second order sampler). I also explain a bit better in code what's going on in kdiffusion.py

I can observe that samplers like Heun, DPM2 a, DPM fast, DPM++ SDE Karras produce very similar results but has a subtle variation from a1111.

I was thinking this would be the sigma values being slightly different. For example:
diffusers: [14.6172, 11.7266,  9.3438,  7.3867,  5.7891,  4.5000,  3.4648,  2.6406,
         1.9912,  1.4834,  1.0908,  0.7910,  0.5649,  0.3965,  0.2732,  0.1842,
         0.1214,  0.0779,  0.0485,  0.0292,  0.0000]

a1111: [14.6116, 11.7413,  9.3685,  7.4192,  5.8284,  4.5394,  3.5029,  2.6763,
         2.0228,  1.5112,  1.1147,  0.8110,  0.5812,  0.4096,  0.2834,  0.1921,
         0.1273,  0.0822,  0.0515,  0.0313,  0.0000]
I used the same method from the k-diffusion pipeline in diffusers to get the min max sigma and I'm not sure how to do it otherwise.

The discrepancy arises from numerical precision. In A1111, betas are generated in float64, and the alphas_cumprod, used in sigma creation, is in float16 format. Conversely, diffusers use float32. torch.cumprod is sensitive to precision, especially when it contains 1000 items. If matching the format, sigmas should be identical. This discrepancy also leads to some differences between A1111 and diffusers' results.

@patrickvonplaten

Ok to close this @patrickvonplaten now that it is in the docs here?

@stevhliu it seems the link is broken. Would you mind resharing a link to where this is in the doc please?

Thank you

You can find it in the Schedulers overview :)

How about the LCM scheduler？How to set init params to align a1111 rst?

@caiqi

@caiqi I am debugging my comfyui implementation of TCD scheduler, and want to print the sigmas array like you did above. Can you please share how you did that? Sorry if it's a basic question, I am still learning my way around diffusers pipelines... I only see timesteps inside the scheduler mixin code and am not sure where to get the sigmas.

UPDATE: I figured it out, added this to the scheduler's set_timesteps method

        self.timesteps = torch.from_numpy(timesteps).to(device=device, dtype=torch.long)
        print(f"timesteps: {repr(self.timesteps)}")
        sigmas = np.sqrt(1 - self.alphas_cumprod)
        print(f"sigmas: {sigmas.to(device=device)[self.timesteps]}")

Hi! Wondering if there's plan for "DPM++ 3M SDE" implementation (or if that's already implemented?)

+1 for this. I'm getting consistent good results with this sampler in A1111, so I'd really love to see it in Diffusers

mark

A1111 / K-Diffusion	Diffusers	Notes
DPM++ 2M	Multistep DPM-Solver
DPM++ 2M Karras	Multistep DPM-Solver	init with `use_karras_sigmas=True`
DPM++ 2M SDE	Multistep DPM-Solver	init with `algorithm_type="sde-dpmsolver++"`
DPM++ 2M SDE Karras	Multistep DPM-Solver	init with `algorithm_type="sde-dpmsolver++"` and `use_karras_sigmas=True`
DPM++ 2S a	N/A	Very similar to DPM++ SDE
DPM++ 2S a Karras	N/A	Very similar to DPM++ SDE Karras
DPM++ SDE	Singlestep DPM-Solver
DPM++ SDE Karras	Singlestep DPM-Solver	init with `use_karras_sigmas=True`
DPM2	DPM Discrete Scheduler
DPM2 Karras	DPM Discrete Scheduler	init with `use_karras_sigmas=True`
DPM2 a	DPM Discrete Scheduler with ancestral sampling
DPM2 a Karras	DPM Discrete Scheduler with ancestral sampling	init with `use_karras_sigmas=True`
DPM adaptive	N/A
DPM fast	N/A	Not really used no?
Euler	Euler scheduler
Euler a	Euler Ancestral Scheduler
Heun	Heun Scheduler
LMS	Linear Multistep
LMS Karras	Linear Multistep	Init with `use_karras_sigmas=True`
N/A	DEIS
N/A	UniPCMultistepScheduler

Sunbelt Computer Software

PL/B Language Development and Support

Navigation Menu

Repository navigation

A1111 <> Diffusers Scheduler mapping #4167

A1111 <> Diffusers Mapping

5 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Sunbelt Computer Software

PL/B Language Development and Support

A1111 <> Diffusers Scheduler mapping #4167

Description

A1111 <> Diffusers Mapping

Activity

patrickvonplaten commented on Jul 20, 2023

stevhliu commented on Jul 20, 2023

rbertus2000 commented on Jul 21, 2023

patrickvonplaten commented on Jul 24, 2023

cmdr2 commented on Jul 24, 2023

cmdr2 commented on Jul 24, 2023

patrickvonplaten commented on Jul 24, 2023

cmdr2 commented on Jul 25, 2023

CapsAdmin commented on Jul 25, 2023

CapsAdmin commented on Jul 26, 2023

CapsAdmin commented on Jul 26, 2023

xhinker commented on Aug 4, 2023

alexisrolland commented on Aug 15, 2023

stevhliu commented on Aug 15, 2023

5 remaining items

patrickvonplaten commented on Sep 12, 2023

patrickvonplaten commented on Sep 12, 2023

CodeCorrupt commented on Sep 13, 2023

jiaxin-lin commented on Dec 9, 2023

patrickvonplaten commented on Dec 18, 2023

caiqi commented on Feb 19, 2024

alexisrolland commented on Feb 19, 2024

stevhliu commented on Feb 20, 2024

abc2cba commented on Mar 4, 2024

dfl commented on Mar 24, 2024

andypotato commented on Mar 27, 2024

Youngon commented on May 21, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions