You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
Currently torch.compile is not compatible with MX data types, mostly because it is not yet clear to me how to cache data so that they are readily available when entering quant_inference_mode.
The main issue arises from the fact that MX data have two views, a groupwise view with an extra group dimension and a compressed view without the extra group dimension (but where the scales are expanded, so much more memory is used).
Describe the solution you'd like
Cache scale/zero_point and all the other quant metadata needed for computation for inference. Not sure what is going to be the best solution.
Additional context
Still not clear the best way to approach this, it might be needed for the proxy to expose some extra helper methods to perform views. Reach out with a proposed solution so we discuss!
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
Currently torch.compile is not compatible with MX data types, mostly because it is not yet clear to me how to cache data so that they are readily available when entering
quant_inference_mode
.The main issue arises from the fact that MX data have two views, a groupwise view with an extra group dimension and a compressed view without the extra group dimension (but where the scales are expanded, so much more memory is used).
Describe the solution you'd like
Cache scale/zero_point and all the other quant metadata needed for computation for inference. Not sure what is going to be the best solution.
Additional context
Still not clear the best way to approach this, it might be needed for the proxy to expose some extra helper methods to perform views. Reach out with a proposed solution so we discuss!
The text was updated successfully, but these errors were encountered: