Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sunshine won't work with Cuda #305891

Open
tymscar opened this issue Apr 22, 2024 · 25 comments
Open

Sunshine won't work with Cuda #305891

tymscar opened this issue Apr 22, 2024 · 25 comments
Labels
0.kind: bug Something is broken 6.topic: cuda Parallel computing platform and API

Comments

@tymscar
Copy link
Contributor

tymscar commented Apr 22, 2024

Describe the bug

Most of the sunshine versions posted on nixpkgs do not work with CUDA enabled.
I have tried manually to pin sunshine to each of the last 20 commits and only around 5 managed to detect my display/encoder at startup.
The nixpkgs revision that works with my machine is 56aeedf456e676d46af2595c3f66de9cf95f2cdd.
That is sunshine v21.0

Steps To Reproduce

Steps to reproduce the behavior:

  1. Install sunshine directly, or using the new service, with pkgs.sunshine.override { cudaSupport = true; };
  2. Run sunshine
  3. Try to connect or go to the homepage where you will see the errors at the top of the page

Expected behavior

I would expect every version to work like the one from revision 56aeedf456e676d46af2595c3f66de9cf95f2cdd where it would detect my gpu, my screen and codecs.

Additional context

The errors I am getting on the homepage are:

Attention! Sunshine detected these errors during startup. We STRONGLY RECOMMEND fixing them before streaming.
- Fatal: Unable to find display or encoder during startup.
- Fatal: Please ensure your manually chosen GPU and monitor are connected and powered on.

Here are the full logs:

[2024:04:22:04:35:40]: Info: Sunshine version: 0.22.2
[2024:04:22:04:35:40]: Error: Couldn't initialize cuda: CUDA_ERROR_SYSTEM_DRIVER_MISMATCH:system has unsupported display driver / cuda driver combination
[2024:04:22:04:35:40]: Info: /dev/dri/card0 -> simpledrm
[2024:04:22:04:35:40]: Warning: No render device name for: /dev/dri/card0
[2024:04:22:04:35:40]: Info: /dev/dri/card1 -> nvidia-drm
[2024:04:22:04:35:40]: Info: System tray created
[2024:04:22:04:35:40]: Error: Environment variable WAYLAND_DISPLAY has not been defined
[2024:04:22:04:35:40]: Info: Detecting monitors
[2024:04:22:04:35:40]: Info: Detected monitor 0: HDMI-0, connected: true
[2024:04:22:04:35:40]: Info: Detected monitor 1: DP-0, connected: true
[2024:04:22:04:35:40]: Info: Detected monitor 2: DP-1, connected: false
[2024:04:22:04:35:40]: Info: Detected monitor 3: DP-2, connected: false
[2024:04:22:04:35:40]: Info: Detected monitor 4: DP-3, connected: true
[2024:04:22:04:35:40]: Info: Detected monitor 5: DP-4, connected: false
[2024:04:22:04:35:40]: Info: Detected monitor 6: DP-5, connected: false
[2024:04:22:04:35:40]: Info: Trying encoder [nvenc]
[2024:04:22:04:35:40]: Info: Screencasting with KMS
[2024:04:22:04:35:40]: Info: /dev/dri/card0 -> simpledrm
[2024:04:22:04:35:40]: Warning: No render device name for: /dev/dri/card0
[2024:04:22:04:35:40]: Info: /dev/dri/card1 -> nvidia-drm
[2024:04:22:04:35:40]: Error: Couldn't find monitor [1]
[2024:04:22:04:35:40]: Info: /dev/dri/card0 -> simpledrm
[2024:04:22:04:35:40]: Warning: No render device name for: /dev/dri/card0
[2024:04:22:04:35:40]: Info: /dev/dri/card1 -> nvidia-drm
[2024:04:22:04:35:40]: Error: Couldn't find monitor [1]
[2024:04:22:04:35:40]: Info: Screencasting with KMS
[2024:04:22:04:35:40]: Info: /dev/dri/card0 -> simpledrm
[2024:04:22:04:35:40]: Warning: No render device name for: /dev/dri/card0
[2024:04:22:04:35:40]: Info: /dev/dri/card1 -> nvidia-drm
[2024:04:22:04:35:40]: Error: Couldn't find monitor [1]
[2024:04:22:04:35:40]: Info: /dev/dri/card0 -> simpledrm
[2024:04:22:04:35:40]: Warning: No render device name for: /dev/dri/card0
[2024:04:22:04:35:40]: Info: /dev/dri/card1 -> nvidia-drm
[2024:04:22:04:35:40]: Error: Couldn't find monitor [1]
[2024:04:22:04:35:41]: Info: Encoder [nvenc] failed
[2024:04:22:04:35:41]: Error: Couldn't find any working encoder matching [nvenc]
[2024:04:22:04:35:41]: Info: // Testing for available encoders, this may generate errors. You can safely ignore those errors. //
[2024:04:22:04:35:41]: Info: Trying encoder [vaapi]
[2024:04:22:04:35:41]: Info: Screencasting with KMS
[2024:04:22:04:35:41]: Info: /dev/dri/card0 -> simpledrm
[2024:04:22:04:35:41]: Warning: No render device name for: /dev/dri/card0
[2024:04:22:04:35:41]: Info: /dev/dri/card1 -> nvidia-drm
[2024:04:22:04:35:41]: Error: Couldn't find monitor [1]
[2024:04:22:04:35:41]: Info: /dev/dri/card0 -> simpledrm
[2024:04:22:04:35:41]: Warning: No render device name for: /dev/dri/card0
[2024:04:22:04:35:41]: Info: /dev/dri/card1 -> nvidia-drm
[2024:04:22:04:35:41]: Error: Couldn't find monitor [1]
[2024:04:22:04:35:41]: Info: Screencasting with KMS
[2024:04:22:04:35:41]: Info: /dev/dri/card0 -> simpledrm
[2024:04:22:04:35:41]: Warning: No render device name for: /dev/dri/card0
[2024:04:22:04:35:41]: Info: /dev/dri/card1 -> nvidia-drm
[2024:04:22:04:35:41]: Error: Couldn't find monitor [1]
[2024:04:22:04:35:41]: Info: /dev/dri/card0 -> simpledrm
[2024:04:22:04:35:41]: Warning: No render device name for: /dev/dri/card0
[2024:04:22:04:35:41]: Info: /dev/dri/card1 -> nvidia-drm
[2024:04:22:04:35:41]: Error: Couldn't find monitor [1]
[2024:04:22:04:35:41]: Info: Encoder [vaapi] failed
[2024:04:22:04:35:41]: Info: Trying encoder [software]
[2024:04:22:04:35:41]: Info: Screencasting with KMS
[2024:04:22:04:35:41]: Info: /dev/dri/card0 -> simpledrm
[2024:04:22:04:35:41]: Warning: No render device name for: /dev/dri/card0
[2024:04:22:04:35:41]: Info: /dev/dri/card1 -> nvidia-drm
[2024:04:22:04:35:41]: Error: Couldn't find monitor [1]
[2024:04:22:04:35:41]: Info: Screencasting with KMS
[2024:04:22:04:35:41]: Info: /dev/dri/card0 -> simpledrm
[2024:04:22:04:35:41]: Warning: No render device name for: /dev/dri/card0
[2024:04:22:04:35:41]: Info: /dev/dri/card1 -> nvidia-drm
[2024:04:22:04:35:41]: Error: Couldn't find monitor [1]
[2024:04:22:04:35:41]: Info: Encoder [software] failed
[2024:04:22:04:35:41]: Fatal: Unable to find display or encoder during startup.
[2024:04:22:04:35:41]: Fatal: Please ensure your manually chosen GPU and monitor are connected and powered on.
[2024:04:22:04:35:41]: Error: Video failed to find working encoder
[2024:04:22:04:35:41]: Info: Adding avahi service Sunshine
[2024:04:22:04:35:41]: Info: Configuration UI available at [https://localhost:47990]
[2024:04:22:04:35:42]: Info: Avahi service Sunshine successfully established.

Notify maintainers

@devusb

Metadata

λ ~/ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.8.6, NixOS, 24.05 (Uakari), 24.05.20240421.f2d7a28`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.2`
 - channels(root): `"nixos-23.11"`
 - nixpkgs: `/nix/store/9hisakick7wdjhafbwmiz6bbhqybkajy-source`

Add a 👍 reaction to issues you find important.

@tymscar tymscar added the 0.kind: bug Something is broken label Apr 22, 2024
@tymscar
Copy link
Contributor Author

tymscar commented Apr 22, 2024

I have tried a reboot as mentioned by #305688 but it does not fix this particular issue.

@tymscar tymscar added the 6.topic: cuda Parallel computing platform and API label Apr 22, 2024
@DieracDelta
Copy link
Member

DieracDelta commented Apr 28, 2024

Reproduced on nixos. Hardware (not sure if it matters) 5950x + 4090 running NixOS unstable. Command used to reproduce:

nix shell -L --expr 'let nixpkgs_r = builtins.getFlake "github:NixOS/nixpkgs/c230c3ecc26a8390b94e7202769365c4ef93b8e2"; in let pkgs = import nixpkgs_r { system = "x86_64-linux"; config = { allowUnfree = true; }; }; in with pkgs; (sunshine.override {cudaSupport = true;})'

then
sudo sunshine

image
image

@luochen1990
Copy link
Contributor

luochen1990 commented May 17, 2024

sunshine.override {cudaSupport = true;} seems changed something, but there is still another problem: with it, the black screen shows some text: "Bluetooth: hci0: No support for _PRR ACPI method"

After reboot, there is still black screen:

$ vainfo --display drm --device /dev/dri/renderD128 |                 grep -E "((VAProfileH264High|VAProfileHEVCMain|VAProfileHEVCMain10).*VAEntrypointEncSlice)|Driver version"
libva info: VA-API version 1.21.0
libva info: Trying to open /run/opengl-driver/lib/dri/nvidia_drv_video.so
libva info: Found init function __vaDriverInit_1_0
libva error: /run/opengl-driver/lib/dri/nvidia_drv_video.so init failed
libva info: va_openDriver() returns 1
vaInitialize failed with error code 1 (operation failed),exit

My Machine:

$ neofetch --stdout
lc@home-pc
----------
OS: NixOS 24.05.20240515.33d1e75 (Uakari) x86_64
Host: ASUSTeK COMPUTER INC. TUF GAMING X670E-PLUS
Kernel: 6.6.30
Uptime: 21 mins
Packages: 2915 (nix-system), 244 (nix-user)
Shell: zsh 5.9
Resolution: 3840x2160
DE: Plasma 5.27.11
WM: KWin
Icons: Papirus-Dark [GTK2/3]
Terminal: .konsole-wrappe
CPU: AMD Ryzen 9 7950X (32) @ 5.000GHz
GPU: AMD ATI Raphael
GPU: NVIDIA GeForce RTX 4090
Memory: 7303MiB / 63429MiB

and, I found that the sunshine nixos module doesn't work with journalctl:

$ journalctl -u sunshine
-- No entries --

@tymscar
Copy link
Contributor Author

tymscar commented May 17, 2024

I think one of the issues with Bluetooth: hci0: No support for _PRR ACPI method we're solved by downgrading to revision 56aeedf456e676d46af2595c3f66de9cf95f2cdd. Can you give that a go @luochen1990 ?

@Garulf
Copy link

Garulf commented Jun 14, 2024

After upgrading from NixOS 23.11 to 24.05 I am seeing the exact same error with sunshine.

Config:

{ config, pkgs, lib, ...}:

let
  sunshineOverride = pkgs.sunshine.override {
    cudaSupport = true;
    stdenv = pkgs.cudaPackages.backendStdenv;
  };
in

with lib;

{

  environment.systemPackages = with pkgs; [
    sunshineOverride
  ];

  services.sunshine = {
    package = sunshineOverride;
    enable = true;
    autoStart = true;
    capSysAdmin = true;
    openFirewall = true;
  };
}

Is this ' following "stdenv" a typo on unstable?

stdenv' = if cudaSupport then cudaPackages.backendStdenv else stdenv;

@Garulf
Copy link

Garulf commented Jun 14, 2024

Suddenly started working after forcing the capture method?

image

image

Haven't fully tested yet.

@tymscar
Copy link
Contributor Author

tymscar commented Jun 14, 2024

Is this ' following "stdenv" a typo on unstable?

No, that's normal. It's a convention called prime notation.

Suddenly started working after forcing the capture method?

That does not work for me sadly.

@vavakado
Copy link

vavakado commented Jul 3, 2024

For me trying to force the capture method worked only on X11 but not wayland

@illustris
Copy link
Contributor

illustris commented Jul 4, 2024

boot.kernelParams = ["nvidia_drm.fbdev=1"] fixed this for me on X11. If your error logs complain about simpledrm, this might work.

@bonsairobo
Copy link
Contributor

bonsairobo commented Aug 8, 2024

For me, forcing the capture method to "KMS" (in the sunshine config) made it so I could at least stream, but it still falls back to CPU encoding instead of using CUDA, which is much slower.

I also tried the Sunshine flatpak (from Flathub), and it works flawlessly with nvenc + CUDA. So it does seem like there's a bug with this nix module.

@Continous
Copy link

Continous commented Aug 10, 2024

For me, forcing the capture method to "KMS" (in the sunshine config) made it so I could at least stream, but it still falls back to CPU encoding instead of using CUDA, which is much slower.

I also tried the Sunshine flatpak (from Flathub), and it works flawlessly with nvenc + CUDA. So it does seem like there's a bug with this nix module.

I can replicate this.

UPDATE: I am now having the same issue when running through Flatpak... This is very strange.

UPDATE 2: I have resolved it no longer working by disabling the open kernel module per issue #334180

@SomeoneSerge
Copy link
Contributor

@Continous can you confirm if #334340 helps

@luochen1990
Copy link
Contributor

luochen1990 commented Aug 26, 2024

I think one of the issues with Bluetooth: hci0: No support for _PRR ACPI method we're solved by downgrading to revision 56aeedf456e676d46af2595c3f66de9cf95f2cdd. Can you give that a go @luochen1990 ?我认为蓝牙的问题之一:hci0:不支持 _PRR ACPI 方法,我们通过降级到修订版 56aeedf456e676d46af2595c3f66de9cf95f2cdd 来解决。你能试一试吗@luochen1990

I'm on 5041afcd2b075aad48f4336668b129cd318edc8b (which includes 56aeedf456e676d46af2595c3f66de9cf95f2cdd) and it is now working, not sure what fixed that.

@Continous
Copy link

@Continous can you confirm if #334340 helps

Currently have some visitors for a couple months so I might be delayed, but I wanted to post to let you know I will test it.

@Continous
Copy link

Continous commented Aug 31, 2024

@Continous can you confirm if #334340 helps

I am presuming unstable has the changes when I say this, however, there has been no improvement. EDIT: Nevermind, it's functional now.

@danilevy1212
Copy link

Hello everyone, I am on the latest unstable and this is 100% still happening, CUDA is not found so sunshine throws this warning: Warning: Attempting to use NVENC without CUDA support. Reverting back to GPU -> RAM -> GPU

This massively slows down encoding on the host. I installed the flatpak and I can confirm this does NOT occur in the flatpak, I get high FPS when streaming from the flatpak. The way sunshine is being packaged is wrong in nixpkgs, the binary does not get dynamically linked to CUDA, that much I could confirm, but I'm no expert so I don't know if that is expected or not.

@danilevy1212
Copy link

I can confirm 100%, sunshine with the cudaSupport = true; override DOES NOT compile with cuda support

sunshine> -- Found Boost: /nix/store/20z0miv70wbl9xvrx4nvq6krrhfl5942-boost-1.81.0-dev/lib/cmake/Boost-1.81.0/BoostConfig.cmake (found version "1.81.0") found components: locale log filesystem program_options
sunshine> -- Looking for a CUDA compiler
sunshine> -- Looking for a CUDA compiler - NOTFOUND

This leads to massive slowdown on capable hardware, looking into the overrideAttrs to see what is missing, I suspect it's the nvcc compiler package.

@SomeoneSerge
Copy link
Contributor

This leads to massive slowdown on capable hardware, looking into the overrideAttrs to see what is missing, I suspect it's the nvcc compiler package.

Indeed I don't see optionals cudaSupport [ cudaPackages.cuda_nvcc ] in nativeBuildInputs. Also it's desirable to remove cudatoolkit in buildInputs/replace it with individual packages. If you open a PR, could you also please add sunshine to pkgs/top-level/release-cuda.nix? Thanks

In future we should be able to detect this kind of mistakes using #351679

@danilevy1212
Copy link

danilevy1212 commented Oct 30, 2024

Ty @SomeoneSerge ! I've been debugging this for a few hours. I'm not really a C++ developer and never worked with CUDA so bare with me, but I hope some of my findings are of use.

If I compile sunshine passing along the cuda_nvcc:

    package = with pkgs;
      (pkgs.sunshine.override {
        cudaSupport = true;
        cudaPackages = cudaPackages;
      })
      .overrideAttrs (old: {
        nativeBuildInputs =
          old.nativeBuildInputs
          ++ [
            cudaPackages.cuda_nvcc
          ];
      });

I still see in the compilation logs:

sunshine> -- Looking for a CUDA compiler
sunshine> -- Looking for a CUDA compiler - NOTFOUND
sunshine> -- Looking for a CUDA host compiler - /nix/store/zxjcb1sdl6rdn6wq6iza8x7ciyj6wnvn-gcc-wrapper-13.3.0/bin/c++

This I believe is what interferes with https://github.com/LizardByte/Sunshine/blob/master/cmake/compile_definitions/linux.cmake#L17a

And I still get the warning [2024:10:30:09:52:09]: Warning: Attempting to use NVENC without CUDA support. Reverting back to GPU -> RAM -> GPU

So I started doing some digging into cmake, the check_language(CUDA) statement and how I could bypass and or populate the vars if fills. I tried this:

      package = (pkgs.sunshine.override {
        cudaSupport = true;
        cudaPackages = cudaPackages;
      })
      .overrideAttrs (old: {
        nativeBuildInputs =
          old.nativeBuildInputs
          ++ [
            cudaPackages.cuda_nvcc
          ];
        cmakeFlags =
          old.cmakeFlags
          ++ [
            "-DCUDA_FAIL_ON_MISSING=ON" # Ensure build fails if CUDA is not fully available
            "-DCMAKE_CUDA_FLAGS=-cudart=shared"
            "-DCMAKE_CUDA_IMPLICIT_LINK_DIRECTORIES=${pkgs.cudaPackages.cudatoolkit}/lib"
            "-DCMAKE_CUDA_COMPILER=${pkgs.cudaPackages.cudatoolkit}/bin/nvcc" # Ensure build fails if CUDA is not fully available
          ];
      });

Which at least by passed the CUDA compiler not found error, but instead I am getting linker errors:

sunshine>   #$ "/nix/store/zxjcb1sdl6rdn6wq6iza8x7ciyj6wnvn-gcc-wrapper-13.3.0/bin"/c++
sunshine>   -D__CUDA_ARCH_LIST__=520 -D__NV_LEGACY_LAUNCH -c -x c++
sunshine>   -DFATBINFILE="\"tmp/a_dlink.fatbin.c\""
sunshine>   -DREGISTERLINKBINARYFILE="\"tmp/a_dlink.reg.c\"" -I.
sunshine>   -D__NV_EXTRA_INITIALIZATION= -D__NV_EXTRA_FINALIZATION=
sunshine>   -D__CUDA_INCLUDE_COMPILER_INTERNAL_HEADERS__ -Wno-psabi
sunshine>   "-I/nix/store/97ba8v5ffpm7r7z8grpaqjwdlsghdqvs-cuda_nvcc-12.4.99/include"
sunshine>   "-I/nix/store/97ba8v5ffpm7r7z8grpaqjwdlsghdqvs-cuda_nvcc-12.4.99/nvvm/include"
sunshine>   -D__CUDACC_VER_MAJOR__=12 -D__CUDACC_VER_MINOR__=4
sunshine>   -D__CUDACC_VER_BUILD__=99 -D__CUDA_API_VER_MAJOR__=12
sunshine>   -D__CUDA_API_VER_MINOR__=4 -D__NVCC_DIAG_PRAGMA_SUPPORT__=1 -m64
sunshine>   "/nix/store/lkxw7rg1mf0c5lrqwf8w8rg2jgjz15l6-cuda-merged-12.4/bin/crt/link.stub"
sunshine>   -o "tmp/a_dlink.o"
sunshine>   #$ "/nix/store/zxjcb1sdl6rdn6wq6iza8x7ciyj6wnvn-gcc-wrapper-13.3.0/bin"/c++
sunshine>   -D__CUDA_ARCH_LIST__=520 -D__NV_LEGACY_LAUNCH -m64 -Wl,--start-group
sunshine>   "tmp/a_dlink.o" "tmp/CMakeCUDACompilerId.o"
sunshine>   "-L/nix/store/lkxw7rg1mf0c5lrqwf8w8rg2jgjz15l6-cuda-merged-12.4/bin/..//lib64/stubs"
sunshine>   "-L/nix/store/lkxw7rg1mf0c5lrqwf8w8rg2jgjz15l6-cuda-merged-12.4/bin/..//lib64""-L/nix/store/97ba8v5ffpm7r7z8grpaqjwdlsghdqvs-cuda_nvcc-12.4.99/nvvm/lib"
sunshine>   -lcudadevrt -lcudart_static -lrt -lpthread -ldl -Wl,--end-group -o "a.out"
sunshine>   /nix/store/vcvhwiilizhijk7ywyn58p9l005n9sbn-binutils-2.43.1/bin/ld: cannot
sunshine>   find -lcudadevrt: No such file or directory
sunshine>   /nix/store/vcvhwiilizhijk7ywyn58p9l005n9sbn-binutils-2.43.1/bin/ld: cannot
sunshine>   find -lcudart_static: No such file or directory
sunshine>   collect2: error: ld returned 1 exit status
sunshine>   # --error 0x1 --

these I believe are some tests that Cmake automatically does whenever it finds a cuda compiler. I'm kinda stuck here, looked into how to add the cudadevrt library with no avail, same with the cudart. Not sure even If im just chasing down a rabbit hole and there is a simple solution.

Would love if someone more experience with CUDA and C++ could help me debug this and get a solution working, at least I am confident that I identified the issue. Maybe there is also a way to patch out the check_language(CUDA) line and pass the relevant flags through cmake flags instead?

I also get these warnings:

sunshine> CMake Warning:
sunshine>   Manually-specified variables were not used by the project:
sunshine>     CMAKE_EXPORT_NO_PACKAGE_REGISTRY
sunshine>     CMAKE_POLICY_DEFAULT_CMP0025
sunshine>     CUDAToolkit_INCLUDE_DIR
sunshine>     CUDAToolkit_ROOT
sunshine>     CUDA_FAIL_ON_MISSING
sunshine>     CUDA_HOST_COMPILER

@SomeoneSerge
Copy link
Contributor

SomeoneSerge commented Oct 30, 2024

cudart_static

Might as well let it use the static library. If you just pass (lib.getDev cuda_cudart) to buildInputs it'll propagate cuda_cudart.static; with .dev you won't have to list all other outputs explicitly; unless sunshine overrides some flags in a way that breaks FindCUDAToolkit.cmake or enable_language(CUDA) this should just work. If it doesn't, you could just add -L${lib.getStatic cuda_cudart}/lib

Instead of ${cudatoolkit}/bin/nvcc best to use (lib.getExe cuda_nvcc)

"-DCMAKE_CUDA_FLAGS=-cudart=shared

I suppose these are nvcc flags. Does it make a difference if you use --cudart over -cudart?

cudadevrt

${lib.getStatic cuda_cudart}/lib/libcudadevrt.a

P.S. Nice work!

@vavakado
Copy link

That's something weird going on, because I get [2024:10:31:00:43:46]: Info: Found H.264 encoder: h264_nvenc [nvenc] [2024:10:31:00:43:46]: Info: Found HEVC encoder: hevc_nvenc [nvenc] using just

  services.sunshine = {
    enable = true;
    openFirewall = true;
    capSysAdmin = true;
    package = (pkgs.sunshine.override { cudaSupport = true; });
  };

in my configuration.

Just in case, here's my nix-info output

❯ nix-shell -p nix-info --run "nix-info -m"
fetching path input 'path:/nix/store/dyzl40h25l04565n90psbhzgnc5vp2xr-source'
 - system: `"x86_64-linux"`
 - host os: `Linux 6.6.57, NixOS, 24.11 (Vicuna), 24.11.20241020.1997e4a`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Lix, like Nix) 2.91.1
System type: x86_64-linux
Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux
Features: gc, signed-caches
System configuration file: /etc/nix/nix.conf
User configuration files: /home/vavakado/.config/nix/nix.conf:/etc/xdg/nix/nix.conf:/home/vavakado/.nix-profile/etc/xdg/nix/nix.conf:/nix/profile/etc/xdg/nix/nix.conf:/home/vavakado/.local/state/nix/profile/etc/xdg/nix/nix.conf:/etc/profiles/per-user/vavakado/etc/xdg/nix/nix.conf:/nix/var/nix/profiles/default/etc/xdg/nix/nix.conf:/run/current-system/sw/etc/xdg/nix/nix.conf
Store directory: /nix/store
State directory: /nix/var/nix
Data directory: /nix/store/cd6zr7wldy90mw628c6gqzf9nshchq23-lix-2.91.1/share`
 - nixpkgs: `/nix/store/dyzl40h25l04565n90psbhzgnc5vp2xr-source`

@danilevy1212
Copy link

danilevy1212 commented Oct 30, 2024

cudart_static

Might as well let it use the static library. If you just pass (lib.getDev cuda_cudart) to buildInputs it'll propagate cuda_cudart.static; with .dev you won't have to list all other outputs explicitly; unless sunshine overrides some flags in a way that breaks FindCUDAToolkit.cmake or enable_language(CUDA) this should just work. If it doesn't, you could just add -L${lib.getStatic cuda_cudart}/lib

Instead of ${cudatoolkit}/bin/nvcc best to use (lib.getExe cuda_nvcc)

"-DCMAKE_CUDA_FLAGS=-cudart=shared

I suppose these are nvcc flags. Does it make a difference if you use --cudart over -cudart?

cudadevrt

${lib.getStatic cuda_cudart}/lib/libcudadevrt.a

P.S. Nice work!

Omg! It worked, TY soo much @SomeoneSerge ! This was so difficult for me to make heads or tails of.

For future reference for other people frustrated by this issue, this is what I am running now:

  # Game-streaming
  services.sunshine = {
    enable = true;
    # Enable nvenc support
    package = with pkgs;
      (pkgs.sunshine.override {
        cudaSupport = true;
        cudaPackages = cudaPackages;
      })
      .overrideAttrs (old: {
        nativeBuildInputs =
          old.nativeBuildInputs
          ++ [
            cudaPackages.cuda_nvcc
            (lib.getDev cudaPackages.cuda_cudart)
          ];
        cmakeFlags =
          old.cmakeFlags
          ++ [
            "-DCMAKE_CUDA_COMPILER=${(lib.getExe cudaPackages.cuda_nvcc)}"
          ];
      });
    openFirewall = true;
    capSysAdmin = true;
  };

Works perfectly! 4K at locked 60 FPS with almost no latency and 1440p at 100+fps.

Not sure how we could upstream this, but definitely there should be a clean way to do it, at least this can help make that happen.

@danilevy1212
Copy link

@vavakado Hey, I think the issue is not that It cannot find the nvenc, is that it was missing some cuda functionality that allowed it to encode the frame without going through RAM, as I understand. Do you get the warning I mentioned in a previous comment?

Warning: Attempting to use NVENC without CUDA support. Reverting back to GPU -> RAM -> GPU

I don't rule out something weird in my config though: https://github.com/danilevy1212/nixos/blob/4afa71d6acd16ea3e292c9e7e6d26cdfad7450d5/hosts/bootse/default.nix#L150-L158 This was my setup when I noticed the issue, I could stream HDR AV1, encoded by nvenc, but it was much slower than other peoples results online with similar hardware, so it made me dig into the logs and that's when I noticed that warning that made me go into this rabbit hole.

@SomeoneSerge
Copy link
Contributor

Not sure how we could upstream this,

Essentially, you implement the same changes in nixpkgs' pkgs/servers/sunshine/default.nix and open a PR

@vavakado
Copy link

@vavakado Hey, I think the issue is not that It cannot find the nvenc, is that it was missing some cuda functionality that allowed it to encode the frame without going through RAM, as I understand. Do you get the warning I mentioned in a previous comment?

Warning: Attempting to use NVENC without CUDA support. Reverting back to GPU -> RAM -> GPU

I don't rule out something weird in my config though: https://github.com/danilevy1212/nixos/blob/4afa71d6acd16ea3e292c9e7e6d26cdfad7450d5/hosts/bootse/default.nix#L150-L158 This was my setup when I noticed the issue, I could stream HDR AV1, encoded by nvenc, but it was much slower than other peoples results online with similar hardware, so it made me dig into the logs and that's when I noticed that warning that made me go into this rabbit hole.

Just went back and checked, and oh yes, I missed this warning when I looked at the output and never noticed any issues with encoding because I only stream to my switch which is 720p.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0.kind: bug Something is broken 6.topic: cuda Parallel computing platform and API
Projects
Status: New
Development

No branches or pull requests

10 participants