RuntimeError: Unrecognized CachingAllocator option: max_split_size_mb=8000

[Eikewi]

Hey,
i have a rtx 3070 ti with 8 gb vram and i tried to just run the WebUI but i always run out of memory before i even can choose which model i want to use. I tried multiple starting options like: --auto-devices --chat --wbits 4 --groupsize 128 --load-in-8bit --no-cache --disk --gpu-memory 8, but nothing seems to work. Has anyone a tip for me?

[Wolvensdale]

You need to read and learn more about each parameters.
There are many unwritten rules.
For example some parameters can't be combined with others. Like if you use autodevices its automatically wont use 8bit,
Or when you use wbits, then its automatically assume that the model you used are quantized one, therefore all other command that are for other model (like ggml, and normal one) won't work. cmiiw.

[adollar47]

Please share your parameters if possible. I'm new to all this and need a place to start from

[Eikewi]

I tried many combinations of commands, but nothing seems to work. I am not even sure if memory is even the problem, because it should ran with the specs i have. It seems more like a problem during Setup. I also search the web for the error but i didn't find anything helpful. Even gpt4 cant help me.

[Besamel]

I got it working with the following options: --auto-devices --chat --model gpt4-x-alpaca-13b-native-4bit-128g --wbits 4 --groupsize 128 --gpu-memory 5 --pre_layer 20

It's a bit slow and dumb though

[Eikewi]

Didn´t work for me either. It seems like it is already failing at the SETUP so maybe it isn´t something about the memory. I uploaded a screenshot of the error maybe it helps understanding the problem.

[Besamel]

Try restarting your machine and trying to launch it before opening anything else. I had no luck before I restarted my machine.

[Eikewi]

Tried it and i tried to reinstall it from scratch, but didnt work either ):

[elcotek]

I use the RTX 2080 8GBvram and --auto-devices --disk --groupsize 128 --load-in-8bit --gpu-memory 3724MiB --pre_layer 25 --chat - Right, i use only 3,724 GB of Vram for the chavinlo_alpaca-native model.

[Eikewi]

Didn´t work either