-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add GPTQModel support for evaluating GPTQ models #2217
base: main
Are you sure you want to change the base?
Conversation
@baberabb Hi, can we get some action on this? What do we need to do to get this reviewed and merged? |
@baberabb Ruff/Lint checks passed. Awaiting review. Thanks. |
@baberabb Ping. Please check our PR. We will push a unit test into |
@baberabb Unit test added. |
Hi! Thanks for the PR, and sorry it took ages for us to review. This looks good to me, but I want to run it through @haileyschoelkopf as well. ps. 3.8 test failing because of a recent |
Add option to use GPTQModel for
lm_eval
. GPTQModel is a replacement for AutoGPTQ on GPTQ quantization and inference with better model support and much faster inference speed out-of-the-box. We have been using it internally with lm-eval for months without issue.