ggllm branch

[kdsull]

Hi Abdeladim, many thanks for this new branch which I didn't expect it done this quick ! I tried on 3 platforms, ie. OSX Mojave, WSL2( ubuntu) and Ubuntu 22.04, but can't make it work... First pip/git install did not work on all three. So I downloaded the project and installed 'python setup.py install', but again all three failed with the same error messages. I attach the error messages for your ref. It obviously is above my understanding, as you've guessed ! I'd appreciate if you could have a look and advise how to make this work on my environment. Cheers !

pyllamacpp_ggllm_errors.txt

[absadiki]

Hi Kidon,

I guess you didn't clone the ggllm submodule. Have you cloned the repo with --recursive ?
Here are the detailed steps :

0. Create and activate a virtual environment ( Create a new one to be on the safe side! ).
1. git clone -b ggllm.cpp --recursive https://github.com/abdeladim-s/pyllamacpp
2. cd pyllamacpp
3. pip install -e .

Let me know if you find any other issue!

[kdsull]

Many thanks Abdeladim. Not so easy as it seemed ! I attach another error msg. On my PC, I do have GPU(1050Ti) but with only 4GB vram, so I'd have to use CPU only, not cublas.

ggllm-errors.txt

[absadiki]

It is OK Kidon,
I only use CPU as well, you can turn off cublas here.

Or .. to make it easy for you, if you are using a Linux x86_64, I have built whl from my machine (I think it will work on your Ubuntu and WSL at least)
I have uploaded it here.

You can install it by running

pip install https://github.com/abdeladim-s/pyllamacpp/releases/download/v2.4.1/pyllamacpp-ggllm-2.4.1-cp310-cp310-linux_x86_64.whl

Let me know if you find any other issues ?

[kdsull]

haha, now it can't NOT work, right ?! I tried two Falcon models (5bit, 8bit) and they both work fine.
Many thanks for your time, and I hope somebody else than me also find this implementation (again very easy, clean, robust one) useful. Just a few prompts tried, but looks like WizardLM models' answers suit me the best. Falcon seems decent enough, but will have to see whether it deserves its fame. Thanks and cheers !!
ps. Unlike the original , it leaves '<|endoftext|>' token at the end of AI's reply before 'You:' prompt.

[absadiki]

you are welcome :)
yeah, I should've build the wheel from the beginning Hahh but glad it worked finally.
yes WizardLM gives good results on my tests as well, I think Falcon gain its fame from its Apache license, or maybe the big 40B model is good (I haven't tried it). Anyways feel free to use whatever suits your needs.

regarding the PS, yes I noticed it leaves '<|endoftext|>' if you are using the CLI, the implementation was quick so I wanted just to make it work. The CLI is just for simple testing, use the generate function to filter out any token you don't want or build your own chat.

Cheers @kdsull!