Repository navigation
How to use dreamshaperXL10_alpha2Xl10.safetensors with controlnet-canny-sdxl-1.0 ? #4618
Description
I want to use dreamshaperXL10_alpha2Xl10.safetensors with controlnet-canny-sdxl-1.0
I downloaded dreamshaperXL10_alpha2Xl10.safetensors file and tried to use :
pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
'./dreamshaperXL10_alpha2Xl10.safetensors',
controlnet=controlnet,
use_safetensors=True,
torch_dtype=torch.float16,
variant="fp16"
)
got error :
pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
File "/opt/conda/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 908, in from_pretrained
cached_folder = cls.download(
File "/opt/conda/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 1330, in download
info = model_info(
File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 110, in _inner_fn
validate_repo_id(arg_value)
File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 158, in validate_repo_id
raise HFValidationError(
huggingface_hub.utils._validators.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': './dream/dreamshaperXL10_alpha2Xl10.safetensors'. Use repo_type argument if needed.
Previously, I tried to use from_single_file insteaed of from_pretrained.
Got error : from_single_file not available with StableDiffusionXLControlNetPipeline.
Please help.
Thanks
Activity
[-]ow to use dreamshaperXL10_alpha2Xl10.safetensors with controlnet-canny-sdxl-1.0 ?[/-][+]How to use dreamshaperXL10_alpha2Xl10.safetensors with controlnet-canny-sdxl-1.0 ?[/+]dai-ichiro commentedon Aug 15, 2023
How about converting dreamshaperXL10_alpha2Xl10.safetensors to diffuser pipeline?
arnold408 commentedon Aug 15, 2023
how to convert it to diffuser pipeline and use it in StableDiffusionXLControlNetPipeline ?
Thanks
dai-ichiro commentedon Aug 15, 2023
Install diffusers from main branch and then see #4461.
How to convert
How to use
arnold408 commentedon Aug 15, 2023
Thanks a lot, I will try this.
sayakpaul commentedon Aug 16, 2023
If you succeeded, could we close the issue?
Killfaeh commentedon Aug 21, 2023
Hello !
The conversion worked for me but not the inference.
I got this error :
Expected tensor for argument #1 'indices' to have one of the following scalar types: Long, Int; but got
torch.HalfTensor instead (while checking arguments for embedding)
Killfaeh commentedon Aug 21, 2023
Here is the code
controlnet = ControlNetModel.from_pretrained('diffusers/controlnet-canny-sdxl-1.0', torch_dtype=torch.float16)
pipe = StableDiffusionXLControlNetPipeline.from_pretrained("./models/Diffuser/dreamshaperXL10_alpha2Xl10", controlnet=controlnet, torch_dtype=torch.float16)
inputImg = Image.open("input/Body-kun-vador-canny.png").convert("RGB")
output = pipe("darth vader holding his light saber in fighting position, trending on art station", negative_prompt='', image=inputImg, num_inference_steps=20, controlnet_conditioning_scale=0.75)
This code works with "stabilityai/stable-diffusion-xl-base-1.0" but not dreamshaperXL10_alpha2Xl10.
sayakpaul commentedon Aug 21, 2023
How did you run the conversion? We cannot reproduce what you posted above.
Killfaeh commentedon Aug 21, 2023
I made a script, here is the code
#!/bin/bash
INPUT_FILE="models/Safetensors/dreamshaperXL10_alpha2Xl10.safetensors"
OUTPUT_DIRECTORY="models/Diffuser/dreamshaperXL10_alpha2Xl10"
python3 diffusers/scripts/convert_original_stable_diffusion_to_diffusers.py --checkpoint_path $INPUT_FILE --dump_path $OUTPUT_DIRECTORY --pipeline_class_name StableDiffusionXLPipeline --from_safetensors
I have the same error when I run an inference with StableDiffusionXLPipeline and from_single_file to load the .safetensors model.
I have tried with the model CounterfaitXL too. Same error.
I have a Mac Intel if this info can help.
sayakpaul commentedon Aug 21, 2023
That script does not support converting SDXL ControlNets.
Killfaeh commentedon Aug 21, 2023
I have copied the command line given by dai-ichiro and replaced the checkpoint_path and dump_path by my own values.
I have checked again, the command line is the same. Is this information wrong ?
Killfaeh commentedon Aug 21, 2023
I found the solution : I have updated some python modules (transformers and accelerate).
The script support SDXL.
sayakpaul commentedon Aug 22, 2023
Cool! Closing this issue then. It would be very nice, if you could post an end-to-end Colab Notebook here. I think that will benefit the community a lot.