Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Solver] Check all output branches are quantized before merging quantizers #2854

Conversation

daniil-lyakhov
Copy link
Collaborator

@daniil-lyakhov daniil-lyakhov commented Jul 30, 2024

Changes

Quantizer merge logic updated to check that all output branches are quantized before quantizers merging and propagating up.

Reason for changes

To prevent merging of quantizers in case of ScaledDotProductAttention op, which should have quantizers on [0, 1] input ports and shouldn't have a quantizer on the 3 input port.

Related tickets

148211
#2766

Tests

  • Common solver test for ScaleDotProductAttention branch merging and quantization initialization
  • Graph tests for torch/ov backends

@daniil-lyakhov daniil-lyakhov requested a review from a team as a code owner July 30, 2024 12:39
@github-actions github-actions bot added NNCF PT Pull requests that updates NNCF PyTorch NNCF Common Pull request that updates NNCF Common NNCF OpenVINO Pull requests that updates NNCF OpenVINO labels Jul 30, 2024
@alexsu52 alexsu52 requested a review from KodiaqQ July 30, 2024 13:08
@daniil-lyakhov
Copy link
Collaborator Author

daniil-lyakhov commented Jul 31, 2024

Scale dot product attention quantization:
Before: quantizers are being merged and propagated up besides the fact that one of the branches is not quantizable
image
After: quantizers are not merged, quantization scheme is correct
image

CC: @KodiaqQ

@daniil-lyakhov daniil-lyakhov force-pushed the dl/scaled_dot_product_attn_prop_fix branch 2 times, most recently from 2645c8c to f9a28e5 Compare July 31, 2024 12:18
@daniil-lyakhov
Copy link
Collaborator Author

bloomz-560m quantization with not fused ScaledDotProductAttention op checked locally

@KodiaqQ KodiaqQ merged commit ef49c75 into openvinotoolkit:develop Aug 5, 2024
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NNCF Common Pull request that updates NNCF Common NNCF OpenVINO Pull requests that updates NNCF OpenVINO NNCF PT Pull requests that updates NNCF PyTorch
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants