exl2 autosplit #6494
exl2 autosplit
#6494
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
edit: posted on issues instead
Im using this model:
https://huggingface.co/LoneStriker/airoboros-70b-3.3-2.4bpw-h6-exl2
If i load it under 1 GPU, it works perfectly with 2k context + 8bit
But if I use autosplit under 2 GPUs as 8k context, the word order gets messed up & missing letters on some words
Is there some option that I should tick to make it work? Just updated to the latest version of text gen and L2 version of airoboros works fine with autosplit on
Beta Was this translation helpful? Give feedback.
All reactions