You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Authors of the paper trained a base controlnet (with a new architecture if I'm not mistaken) on 9 different conditions to allow finetuning on new conditions easily using a LoRA rank 128. This method allows finetuning a novel condition using less than 1000 examples with less than 24GB of VRAM in a few hours.
The potential is really high and for having trained a new unseen condition myself, I can confirm it works pretty well (even though my dataset has 5K examples but I did train it quickly using only a 3090 GPU).
The only problem is that the training and inference code seems to be done on the old stable diffusion code and it may be difficult to port it to diffusers.
Anyone interested in implementing the training and inference code in diffusers ? License is Apache-2.0 license
Open source status
The model implementation is available.
The model weights are available (Only relevant if addition is not a scheduler).
Model/Pipeline/Scheduler description
Authors of the paper trained a base controlnet (with a new architecture if I'm not mistaken) on 9 different conditions to allow finetuning on new conditions easily using a LoRA rank 128. This method allows finetuning a novel condition using less than 1000 examples with less than 24GB of VRAM in a few hours.
The potential is really high and for having trained a new unseen condition myself, I can confirm it works pretty well (even though my dataset has 5K examples but I did train it quickly using only a 3090 GPU).
The only problem is that the training and inference code seems to be done on the old stable diffusion code and it may be difficult to port it to diffusers.
Anyone interested in implementing the training and inference code in diffusers ? License is Apache-2.0 license
Open source status
Provide useful links for the implementation
Ctrlora repo : https://github.com/xyfJASON/ctrlora
Ctrlora paper : https://arxiv.org/abs/2410.09400
The text was updated successfully, but these errors were encountered: