iframe-proxy

sayakpaul · 2023-07-30T11:00:53Z

This PR fixes the LoRA loading part for the text encoders, especially the ones we see for SDXL 0.9.

Additionally, it also refactors some of the loading logic to help make the state_dict parsing easier.

I tested with the following:

from diffusers import DiffusionPipeline, DPMSolverMultistepScheduler
import safetensors
import torch 

base_model_id = "stabilityai/stable-diffusion-xl-base-0.9"
pipeline = DiffusionPipeline.from_pretrained(base_model_id, torch_dtype=torch.float16).to("cuda:0")
pipeline.scheduler = DPMSolverMultistepScheduler.from_config(
    pipeline.scheduler.config, algorithm_type="sde-dpmsolver++"
)
pipeline.load_lora_weights(".", weight_name="daiton.safetensors")

prompt = "alien, monster, daiton, high resolution, 4k, photorealistic, in a spaceship, gloomy"
generator = torch.manual_seed(22)
num_inference_steps = 50
guidance_scale = 5

image = pipeline(
    prompt=prompt, num_inference_steps=num_inference_steps,
    generator=generator, guidance_scale=guidance_scale
).images[0]
image.save("image.png")

And got:

LoRA is from here: https://civitai.com/models/108448/daiton-sdxl-test

@patrickvonplaten I see we don't do SLOW tests for SDXL, probably because we run the SLOW tests on a V100, which might fall short of running SDXL. I am bringing it up because I wanted to add a couple of slow tests to test_lora_layers.py to test the SDXL LoRAs from CivitAI.

Some comments are in line.

Also, I have run the SLOW tests and they pass successfully.

sayakpaul · 2023-07-30T11:04:13Z

+        if is_network_alphas_populated and len(network_alphas) > 0:
+            raise ValueError(
+                f"The `network_alphas` has to be empty at this point but has the following keys \n\n {', '.join(network_alphas.keys())}"
+            )


Like: https://github.com/huggingface/diffusers/pull/4371/files#r1278549857

Can we show ValueError as warning log (non supported lora ) and move on instead of raise error?

I think a ValueError is great here

HuggingFaceDocBuilderDev · 2023-07-30T11:07:43Z

The documentation is not available anymore as the PR was closed or merged.

adhikjoshi · 2023-07-31T13:33:44Z

Lycoris still not working for SDXL

https://civitai.com/models/108011/fcbodybuildingxl-10-for-sdxl

sayakpaul · 2023-07-31T13:37:41Z

Lycoris still not working for SDXL

https://civitai.com/models/108011/fcbodybuildingxl-10-for-sdxl

I think we talked about it in the other thread too. It might be happening because LyCORIS adapters can contain LoHA modules too which we don't support yet.

Let's try not to duplicate these things :-)

patrickvonplaten

Very nice! Only left a couple of minor suggestions

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

sayakpaul · 2023-08-02T11:30:51Z

* temporarily disable text encoder loras. * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debbuging. * modify doc. * rename tests. * print slices. * fix: assertions * Apply suggestions from code review Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> --------- Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

sayakpaul added 26 commits July 30, 2023 11:24

temporarily disable text encoder loras.

ebcb4c9

debugging

1c2aaa9

debugging

123c8d7

debugging

dfbbc34

debugging

7245680

debugging

2a66bc7

debugging

9713b12

debugging

71c088f

debugging

2be787e

debugging

9fe54e1

debugging

4c7fa7b

debugging

ad667a1

debugging

6d37a12

debugging

1e44e98

debugging

bde02e6

debugging

0abf1bf

debugging

3b3d480

debugging

df03ca2

debugging

7e2871c

debugging

8406a2c

debugging

eb4e851

debbuging.

cd2c001

modify doc.

5c79a81

rename tests.

b82c1fd

print slices.

de0b4bf

fix: assertions

2a44d2d

sayakpaul requested a review from patrickvonplaten July 30, 2023 11:00