Repository navigation
Ctrlora implementation : Easy solution to train a controlnet with < 1000 examples #9713
Description
Model/Pipeline/Scheduler description
Authors of the paper trained a base controlnet (with a new architecture if I'm not mistaken) on 9 different conditions to allow finetuning on new conditions easily using a LoRA rank 128. This method allows finetuning a novel condition using less than 1000 examples with less than 24GB of VRAM in a few hours.
The potential is really high and for having trained a new unseen condition myself, I can confirm it works pretty well (even though my dataset has 5K examples but I did train it quickly using only a 3090 GPU).
The only problem is that the training and inference code seems to be done on the old stable diffusion code and it may be difficult to port it to diffusers.
Anyone interested in implementing the training and inference code in diffusers ? License is Apache-2.0 license
Open source status
- The model implementation is available.
- The model weights are available (Only relevant if addition is not a scheduler).
Provide useful links for the implementation
Ctrlora repo : https://github.com/xyfJASON/ctrlora
Ctrlora paper : https://arxiv.org/abs/2410.09400
Activity
sayakpaul commentedon Nov 1, 2024
I think this could just be about:
ControlNetModelfrom the pretrained UNet with thefrom_unet()method.add_adapter()on the model with the respective LoRA params. For thisControlNetModelneeds to also subclass fromPeftLoaderMixin.The pre-trained LoRAs are here:
https://huggingface.co/xyfJASON/ctrlora/tree/main/ctrlora-loras/base-conditions
So, for inference and support loading these,
ControlNetModeljust like we do for the UNet:diffusers/src/diffusers/loaders/unet.py
Line 354 in 9dcac83
Would you maybe like to give this a try? Happy to guide.
Cc: @asomoza
github-actions commentedon Nov 25, 2024
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
jloveric commentedon Feb 4, 2025
Is this being implemented as part of this PR? #10686
lavinal712 commentedon Feb 5, 2025
Ctrlora has new modules that ControlNet cannot support.
github-actions commentedon Mar 2, 2025
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.