Anyone know an Alpaca 13b or 30b +4bit models?

[JustinHinh]

I've managed to get the Alpaca Loras [7b, 13b, 30b] just fine on my RTX 3080 10gb

I also got this native version of Alpaca 7b and Alpaca native 4-bit working

I'm wondering if there are any 13b or 30b native + 4bit models out there?

Amazing that this stuff can run on my hardware

[St33lMouse]

I got native alpaca 7b to work without trouble on ooba. Currently looks like 4bits is borked in the latest update. I'd be very interested to get a 13b native or 4bit (for when we fix 4bits here).

Alpaca does a better job figuring out what you want and delivering than anything else I've seen, so it will be awesome to see it running.

[nullgate]

No, peft lora is not compatible with QuantLinear used by GPTQ-for-LLaMA.
You have to wait for someone to make a patch.
You can use --load-in-8bit for now, and it works very with alpaca 13b lora.

Below is an instruction that describes a task. Write a response that appropriately completes the request.

Instruction:

'The school bus passed the racecar because it was driving so quickly.' In the previous sentence, what was driving so quickly?

Response:

The school bus was driving quickly in order to pass the race car.

[Ph0rk0z]

works fine for me through https://github.com/johnsmith0031/alpaca_lora_4bit

I tried 7b, 13b and 30b.. about to try opt loras if I find any. I made it seamless on my fork but it's still using the old GPTQ.

[BetaDoggo]

Native finetunes of the 13b+ models probably won't happen for a while because they require at least 8xa100s with current trainers and most people can't afford that.

Edit: Dep did it https://huggingface.co/chavinlo/alpaca-13b

[Ph0rk0z]

Not native but:
here https://huggingface.co/chansung/alpaca-lora-30b/tree/main

https://huggingface.co/baruga/alpaca-lora-13b/tree/main

and someone combined it with the model : https://huggingface.co/elinas/alpaca-13b-lora-int4/tree/main

The way to do this would be to merge the lora into the model and quantize it to 4bit on your own.

[BetaDoggo]

There is a native 13b finetune now, you will need to quantize it though.
https://huggingface.co/chavinlo/alpaca-13b