Best way to run GPT4-x-Alpaca-13B on 3060TI / 16GB ram?

[Nightnightlight]

I can just barely run it with default settings from the installer at 1.09 tokens a second. Anyway to perhaps improve those speeds or is that the best my setup is gonna get?

[knoopx]

you sure you're using GPU inference? llama.cpp-based (ggml) models run on CPU.

[Nightnightlight]

Apparently not looking at performance tab even though I selected the nvidia option on the installer. Tried the --pre_layer 30 option as well and it just runs out of memory then mid generation.

[colaborat0r]

How much VRAM?
My 3060 is 12GB VRAM, 32GB DDR3 ram.
This model is the best performance I've gotten so far.

[flesnuk]

using --wbits 4 --groupsize 128 --pre_layer 30 --auto-launch with this 4bit model https://huggingface.co/anon8231489123/gpt4-x-alpaca-13b-native-4bit-128g
I get 0.4-0.6 tok/s with your exact same gpu and ram

[devSarry]

My results with the same setting 3060ti 32GB ram
Output generated in 60.60 seconds (1.24 tokens/s, 75 tokens, context 49)