Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimization for ConvNd if dropout=0. #2371

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

gslama12
Copy link

@gslama12 gslama12 commented Feb 8, 2025

As discussed in #2153

Copy link
Member

@BenjaminBossan BenjaminBossan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this DoRA optimization to conv layers. Generally, this looks good, but I have one comment about the bias term.

result_dora = (mag_norm_scale - 1) * (
self.conv_fn(

if base_result is None:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wondering: Do we not need to account for the possibility that the base layer has a bias term, same as we do for linear?

if base_result is not None:
bias = base_layer.bias
if bias is not None:
base_result = base_result - bias

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants