You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This seems to be the more sensible approach, but will probably not make much difference since the ControlNet can also learn the shift. It might speed up convergence slightly.
Best,
Tobias
Reproduction
Train an SD3 ControlNet and during log_validation it will be executed.
I have tried train sd3 controlnet, but it seems the validation results are really bad, and the training loss was oscillating all the time, you can take a look the results at this discussion #9675
Maybe you have any suggestions to make training sd3 controlnet have better results? thank you!
Could you please describe the bug? Maybe I have same bug like you
controlnet_pooled_projections variable is different at inference and training time,when training use pooled_prompt_embeds,inference use torch.zeros_like(pooled_prompt_embeds)
Describe the bug
Hi,
I think I found an issue that causes a misalignment between training and inference in SD3 ControlNet.
diffusers/src/diffusers/pipelines/controlnet_sd3/pipeline_stable_diffusion_3_controlnet.py
Line 977 in a3e8d3f
I think the if-else block starting there is not correct. It should be
Given that in training, the pooled_prompt_embeds are fed to the model:
diffusers/examples/controlnet/train_controlnet_sd3.py
Line 1293 in a3e8d3f
Additionally, I am wondering if this line:
diffusers/examples/controlnet/train_controlnet_sd3.py
Line 1287 in a3e8d3f
Should be aligned with this line:
diffusers/examples/controlnet/train_controlnet_sd3.py
Line 1257 in a3e8d3f
This seems to be the more sensible approach, but will probably not make much difference since the ControlNet can also learn the shift. It might speed up convergence slightly.
Best,
Tobias
Reproduction
Train an SD3 ControlNet and during log_validation it will be executed.
Logs
No response
System Info
diffusers==0.30.3
Who can help?
@yiyixuxu @sayakpaul
The text was updated successfully, but these errors were encountered: