Let embedding model run on GPU

[ThiloteE]

Historical "what the fuck" is available at JabRef#11430 (comment)

Advantages:

• For LLMs, GPU is much faster than CPU. 10 times + X faster, depending on hardware.

Disadvantages:

• Please correct me, if I am wrong, but I expect dependencies for GPU backend are required. (E.g. llama.cpp, Nvidia (drivers, Cuda toolkit libraries), Vulkan, RoCm, SYCL, ...)

[ThiloteE]

If implemented, let users choose backend and hardware (CPU vs GPU / GPU1 or GPU2 or GPU3 ...) choose in preferences.

[InAnYan]

Currently in langchain4j in-process embedding models (meaning they run locally on a computer) are run only on CPU. There is an issue to run embedding models on GPU, but it's not resolved.

In order to implement this we have these choices:

1. Wait for implementation in langchain4j: simpler to develop, better from architectural point of view.
2. Write fix ourselves for langchain4j: good.
3. Use external modules and write all the support code ourselves in JabRef: the fastest way

It's a very good idea, we should look into it, but probably a bit later, when we finally release AI chat and, maybe, add summarization.

I'll mark the issue as low-priority, but it's only low priority for this context: week 1 and first release

[InAnYan]

Actually, no, I'll remove low-priority, and won't assign a milestone

[koppor]

I collect it at the final "anything else" Milestone "final polishing" 😅

[ThiloteE]

GPU support (for embedding models) with llama.cpp:

• tutorial : compute embeddings using llama.cpp ggerganov/llama.cpp#7712
• https://github.com/kherud/java-llama.cpp?tab=readme-ov-file#custom-llamacpp-setup-gpu-acceleration

[ThiloteE]

GPU support with Deep Java library: https://docs.djl.ai/engines/onnxruntime/onnxruntime-engine/index.html#install-gpu-package. Unfortunately they also use Microsofts ONNX, which seems to be very slow. I assume models need to be compatible with ONNX too, because not many models are uploaded on Huggingface in ONNX file format!

[koppor]

At least, one can paint everything blue in the CPU utilization

[ThiloteE]

One solution to providing support for GPU acceleration for LLMs (NOT necessarily embedding models!) is to provide proper support for OpenAI API. See issue JabRef#11872. Using external applications like llama.cpp, GPT4All, LMStudio, Ollama, Jan, KobolCPP etc. that already provide support for GPU acceleration, there is no need to add and maintain this feature in JabRef. It would still be nice to have GPU acceleration for embedding models though. Maybe do it like Koboldcpp and only provide a Vulkan backend, which is much much smaller than a Cuda backend (~1.5 GB in pytorch; 200 - 500 MB in llama.cpp).