[BUG]: Failed to load ./runtimes/win-x64/native/cuda12/llama.dll

[vadsavin]

Description

Installed CUDA12 toolkit:
C:\Users\Home>nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Wed_Oct_30_01:18:48_Pacific_Daylight_Time_2024
Cuda compilation tools, release 12.6, V12.6.85
Build cuda_12.6.r12.6/compiler.35059454_0

Installed packeges:

• LLamaSharp.Backend
• LLamaSharp.Backend.Cuda12
• LLamaSharp.Backend.Cuda12.Windows

Run:
using LLama.Native;
NativeLibraryConfig.All
.WithCuda()
.SkipCheck(true)
.WithAutoFallback(false)
.WithLogCallback((level, message) => Console.Write($"{level}: {message}"));
NativeApi.llama_empty_call();

Outputs:

Debug: Loading library: 'llama'
Info: Detected OS Platform: 'WINDOWS'
Debug: Detected OS string: 'win-x64'
Debug: Detected extension string: '.dll'
Debug: Detected prefix string: ''
Info: NativeLibraryConfig Description:
- LibraryName: LLama
- Path: ''
- PreferCuda: True
- PreferVulkan: True
- PreferredAvxLevel: NoAVX
- AllowFallback: False
- SkipCheck: True
- SearchDirectories and Priorities: { ./ }
Info: NativeLibraryConfig Description:
- LibraryName: LLama
- Path: ''
- PreferCuda: True
- PreferVulkan: True
- PreferredAvxLevel: NoAVX
- AllowFallback: False
- SkipCheck: True
- SearchDirectories and Priorities: { ./ }
Debug: Got relative library path 'runtimes/win-x64/native/cuda12/llama.dll' from local with (NativeLibraryName: LLama, UseCuda: True, UseVulkan: False, AvxLevel: None), trying to load it...
Debug: Found full path file './runtimes/win-x64/native/cuda12/llama.dll' for relative path 'runtimes/win-x64/native/cuda12/llama.dll'
Info: Failed Loading './runtimes/win-x64/native/cuda12/llama.dll'
Debug: Got relative library path 'runtimes/win-x64/native/vulkan/llama.dll' from local with (NativeLibraryName: LLama, UseCuda: False, UseVulkan: True, AvxLevel: None), trying to load it...
Debug: Found full path file 'runtimes/win-x64/native/vulkan/llama.dll' for relative path 'runtimes/win-x64/native/vulkan/llama.dll'
Info: Failed Loading 'runtimes/win-x64/native/vulkan/llama.dll'
Unhandled exception. System.TypeInitializationException: The type initializer for 'LLama.Native.NativeApi' threw an exception.
---> LLama.Exceptions.RuntimeError: Failed to load the native library. Please check the log for more information.
at LLama.Native.NativeLibraryUtils.TryLoadLibrary(NativeLibraryConfig config, INativeLibrary& loadedLibrary)
at LLama.Native.NativeApi.<>c.b__50_0(String name, Assembly _, Nullable`1 _)
at System.Runtime.InteropServices.NativeLibrary.LoadLibraryCallbackStub(String libraryName, Assembly assembly, Boolean hasDllImportSearchPathFlags, UInt32 dllImportSearchPathFlags)
at LLama.Native.NativeApi.llama_max_devices()
at LLama.Native.NativeApi.llama_empty_call()
at LLama.Native.NativeApi..cctor()
--- End of inner exception stack trace ---
at LLama.Native.NativeApi.llama_empty_call()

And it works fine (slow) when i use CPU backend. So what can I do to run model on GPU with CUDA?

Reproduction Steps

run example code

Environment & Configuration

• Operating system: win10
• .NET runtime version: net8
• LLamaSharp version: 0.19.0
• CUDA version (if you are using cuda backend): 12.6
• CPU & GPU device: i7-960 x GTX970

Known Workarounds

No response

[martindevans]

It's finding the CUDA binaries but failing to load them for some reason, probably a prerequisite is missing.

[vadsavin]

What kind of prerequisite should I check for?

[martindevans]

Th CUDA toolkit is the most common one people are missing (https://developer.nvidia.com/cuda-toolkit)

[vadsavin]

First line of subject tells that CUDA Toolkit is installed, isn't it?

Also -- LLama.Exceptions.RuntimeError: Failed to load the native library. Please check the log for more information. : how can I retrieve logs?

[martindevans]

First line of subject tells that CUDA Toolkit is installed, isn't it?

Oops, missed that! Unfortunately that makes it a lot trickier to debug, something like dependency walker might help (but it's hard to use).

how can I retrieve logs?

That log message is coming from here, it's referring to the logs you're already looking at.