Building with CuBLAS without AVX2

[hailtotheking424]

I have an older processor, AMD FX-6350, and a newer graphics card, a Nvidia GTX 980 TI. Obviously I want to use the GPU, so I've installed cudatoolkit and I've edited the makefile lightly:

#if not on windows, they are clearly building it themselves, so lets just use whatever is supported
CFLAGS += -march=athlon-fx -mtune=native

and also at

NVCCFLAGS += -arch=sm_52

to get to this point. Now when I try to build, it returns

nvcc --forward-unknown-to-host-compiler -use_fast_math -arch=sm_52 -DGGML_CUDA_DMMV_X=32 -DGGML_CUDA_MMV_Y=1 -DK_QUANTS_PER_ITERATION=2 -DGGML_CUDA_MMQ_Y=64 -I. -I./common -I./include -I./include/CL -I./otherarch -I./otherarch/tools -O3 -DNDEBUG -std=c++11 -fPIC -DLOG_DISABLE_LOGS -D_GNU_SOURCE -pthread -s -Wno-multichar -Wno-write-strings -pthread -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -Wno-pedantic -c ggml-cuda.cu -o ggml-cuda.o
/usr/include/c++/11.2.0/type_traits(1356): error: type name is not allowed

/usr/include/c++/11.2.0/type_traits(1356): error: type name is not allowed

/usr/include/c++/11.2.0/type_traits(1356): error: identifier "__is_same" is undefined

ggml-cuda.cu(601): warning: function "warp_reduce_sum(half2)" was declared but never referenced

ggml-cuda.cu(622): warning: function "warp_reduce_max(half2)" was declared but never referenced

3 errors detected in the compilation of "/tmp/tmpxft_00005394_00000000-6_ggml-cuda.cpp1.ii".
make: *** [Makefile:187: ggml-cuda.o] Error 1

These are too arcane especially in combination to return good search results so I have to resort to asking if anyone knows where the next place I should look is. I'm running Slackware 15

[LostRuins]

Does the unmodified makefile not work? It should attempt to use the native arch and all natively support intrinsics, so self compile shouldn't be a problem.

[LostRuins]

May be a problem with your CUDA toolkit. What version did you install?

How about trying to build via conda instead? Use the koboldcpp.sh script which downloads and sets up all dependencies for you.

[hailtotheking424]

bash-5.1$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Oct_23_19:24:38_PDT_2019
Cuda compilation tools, release 10.2, V10.2.89

I suspect it may be a distro specific issue with how CUDA is installed. Will remove and try with conda

[hailtotheking424]

Alright, that builds but when I tried to launch a model it just exited. Running koboldcpp.py works and shows CuBLAS as an option, but when exits with "Illegal Instruction" when I try to launch a model. I'm afraid this may be exactly where I started hence my concern about it being related to AVX2 and editing of the CPU type designation in the makefile.

If you have any ideas about how to make this method work I'm all ears and greatly appreciate your time, otherwise I figure the only next step is to try and deal with the weird, frankly incorrect way the Slackbuild installs cudatoolkit

[LostRuins]

Okay you can retry the conda build with a small modification.

Go to the makefile and edit line at https://github.com/LostRuins/koboldcpp/blob/concedo/Makefile#L122

CFLAGS += -mavx2 -msse3 -mfma -mf16c -mavx

replace it with

CFLAGS += -march=native -mtune=native

Then run koboldcpp.sh again. This should prevent illegal instruction errors.

[hailtotheking424]

That worked! I suppose it makes sense reading over the makefile but it's not something I would have figured out on my own. Much appreciation

[navr32]

Can someone add a build config to have old cpu without avx or avx2 build with the vulkan bakend support and the cuda backend. ?
I use them on llamacpp and they work without avx or avx2.
many Thanks.

[mlgitter]

I've just asked a similar question.

#1068

[LostRuins]

This is now possible, grab the old cpu exe:

https://github.com/LostRuins/koboldcpp/releases/latest/download/koboldcpp_oldcpu.exe

comes with noavx2 cublas and noavx2 vulkan.

[mlgitter]

This is now possible, grab the old cpu exe:

https://github.com/LostRuins/koboldcpp/releases/latest/download/koboldcpp_oldcpu.exe

comes with noavx2 cublas and noavx2 vulkan.

I've tried the latest release koboldcpp_oldcpu.exe and unfortunately it ended up with exception for both cublas and vulkan.

Traceback (most recent call last):
File "koboldcpp.py", line 4299, in
main(parser.parse_args(),start_server=True)
File "koboldcpp.py", line 3974, in main
loadok = load_model(modelname)
File "koboldcpp.py", line 817, in load_model
ret = handle.load_model(inputs)
OSError: [WinError -1073741795] Windows Error 0xc000001d
[10956] Failed to execute script 'koboldcpp' due to unhandled exception!

Please see the details in my similar topic
#1068 (reply in thread)