Enabling GPU on NVIDIA drivers causes crashes

[JJ-IsShort]

Useful information:

Please, try to gather as much of useful information as possible and follow
these instructions:

• Version:

  • Run this command: wayvnc -V
  • wayvnc: v0.6.2-30-g945f590 (master)
  • neatvnc: v0.6.0-31-g56f1c12 (master)
  • aml: v0.3.0-1-gede4249 (master)

• Try to reproduce while capturing a trace log:

  • wayvnc -Ltrace | tee wayvnc-crash.log

TRACE: ../src/main.c: 234: Registering new wlr_output_power_manager
TRACE: ../src/main.c: 224: Registering new xdg_output_manager
TRACE: ../src/main.c: 205: Registering new output 49
TRACE: ../src/output.c: 222: Output 49 name: HDMI-A-1
TRACE: ../src/output.c: 231: Output 49 description: Samsung Electric Company LS24AG30x H4PR700788 (HDMI-A-1)
TRACE: ../src/output.c: 274: Output HDMI-A-1 power state changed to ON
Info: Capturing output HDMI-A-1
Info: >> Samsung Electric Company LS24AG30x H4PR700788 (HDMI-A-1) 1920x1080+0x0 Power:ON
DEBUG: ../subprojects/neatvnc/src/server.c: 1360: Trying address: 0.0.0.0
DEBUG: ../subprojects/neatvnc/src/server.c: 1380: Successfully bound to address
Info: Listening for connections on 0.0.0.0:25565
DEBUG: ../src/ctl-server.c: 751: Initializing wayvncctl socket: /run/user/1000/wayvncctl
Info: New client connection from <redacted>: 0x5559d9eac8c0 (ref 1)
Info: Starting screen capture
DEBUG: ../src/main.c: 1100: Client connected, new client count: 1
DEBUG: ../src/ctl-server.c: 879: Enqueueing client-connected event: {"id":"1","hostname":"<redacted>","username":null,"seat":"seat0","connection_count":1}
DEBUG: ../src/ctl-server.c: 906: Enqueued client-connected event for 0 clients
Info: Choosing zrle encoding for client 0x5559d9eac8c0

• Get the stack trace:
  • If have coredumpctl, you can gather the stack trace after a crash using
    coredumpctl gdb wayvnc and then run bt full to obtain the stack trace.
  • Otherwise, you can either locate the core file and load it into gdb or run
    wayvnc in gdb like so:
    • gdb --args wayvnc -Ltrace
  • If the lines mentioning wayvnc, neatvnc or aml have ??, please compile
    wayvnc and those other projects from source with debug symbols and try
    again.

#0  0x00007f8648a8e83c in  () at /usr/lib/libc.so.6
#1  0x00007f8648a3e668 in raise () at /usr/lib/libc.so.6
#2  0x00007f8648a264b8 in abort () at /usr/lib/libc.so.6
#3  0x00007f8648a263dc in  () at /usr/lib/libc.so.6
#4  0x00007f8648a36d26 in  () at /usr/lib/libc.so.6
#5  0x00007f8649016dc3 in zrle_encoder_do_work (obj=0x5559d9ec23e0) at ../subprojects/neatvnc/src/zrle.c:354
        self = 0x5559d9eec440
        rc = -1
        fb = 0x5559d9eb9460
        __PRETTY_FUNCTION__ = "zrle_encoder_do_work"
        buffer_size = 0
        dst = {data = 0x0, len = 0, cap = 0}
        src_fmt = {bits_per_pixel = 32 ' ', depth = 24 '\030', big_endian_flag = 0 '\000', true_colour_flag = 1 '\001', red_max = 255, green_max = 255, blue_max = 255, red_shift = 16 '\020', green_shift = 8 '\b', blue_shift = 0 '\000', padding = "\177\000"}
#6  0x00007f8649036904 in worker_fn (context=0x0) at ../subprojects/aml/src/thread-pool.c:111
        work = 0x5559d9ebe3a0
        cb = 0x7f8649016c2c <zrle_encoder_do_work>
        aml = 0x0
        ss = {__val = {18446744067267035135, 0 <repeats 15 times>}}
        __PRETTY_FUNCTION__ = "worker_fn"
#7  0x00007f8648a8c9eb in  () at /usr/lib/libc.so.6
#8  0x00007f8648b10dfc in  () at /usr/lib/libc.so.6

• Describe how to reproduce the problem
  I applied the patch recommended in wayvnc doesn't work with nvidia drivers, because neatvnc doesn't support 24bit pixel input #139 to wlroots in my Hyprland DE. I then had the issue from Segmentation Fault on Arch/Wayland/Hyprland #248 and applied the patch from there to neatvnc. I was able to connect using RealVNC to Hyprland running on a PC with an NVIDIA card. It was in CPU mode so I added --gpu to it, and this error happened.

[any1]

Coercing the software rendering pixel format to a 32 bit one instead of 24 will not change the GPU buffer format into a 32 bit one as well. It's still 24 bits, and those are not currently implemented. Besides, if you're using ZRLE encoding, you won't notice any performance improvement from setting the --gpu flag.

[JJ-IsShort]

Oh well, looks like to make this work, I will have to make a lot of changes. I do not guarantee I will be able to even do it, and I don't guarantee it will be good, but if it works, I could try to make some commits.

[maztheman]

On my machine, nvidia proprietary driver, gbm_bo_map() returns ENOSYS which means its not implemented.

[biergaizi]

Besides, if you're using ZRLE encoding, you won't notice any performance improvement from setting the --gpu flag.

You have been misled by the log message. The ZRLE log message was the last words before WayVNC crashed, but the actual crash occurred the H.264 codepath before it was able to print any debug log. I believe I've now debugged the problem, and it should affect both Nvidia and AMD GPUs, see Issue #327 for details.

[any1]

You have been misled by the log message. The ZRLE log message was the last words before WayVNC crashed,

I don't think so. If the client chooses ZRLE, wayvnc is not going to even try H.264.

[biergaizi]

If the client chooses ZRLE, wayvnc is not going to even try H.264.

WayVNC doesn't try H.264, but TigerVNC does. For the record, TigerVNC appears to always try ZRLE first, before it immediately tries again via H.264 (Even if H.264 is set as the preferred option). WayVNC only logs Choosing open-h264 encoding for client to the console after H.264 availability check succeeds. If the H.264 check itself crashes, the debug log would look misleading, as if the ZRLE encoding causes a H.264 crash.

See the two highlighted lines.

Info: Capturing output HEADLESS-1
Info: >> Headless output 1 1280x720+0x0 Power:UNKNOWN
Info: Listening for connections on 10.3.0.2:5900
Info: New client connection from 10.3.0.1: 0x55aa74ef3bb0 (ref 1)
Info: Starting screen capture
-----> Info: Choosing zrle encoding for client 0x55aa74ef3bb0
Info: Opened DRM device /dev/dri/renderD128: driver amdgpu version 3.57.0.
Info: w:1 h:1 pixfmt:drm_prime tb:1/1 fr:0/1 sar:0/1
Info: libva: VA-API version 1.22.0
Info: libva: Trying to open /usr/lib64/va/drivers/radeonsi_drv_video.so
Info: libva: Found init function __vaDriverInit_1_22
Info: libva: va_openDriver() returns 0
Info: Initialised VAAPI connection: version 1.22
Info: VAAPI driver: Mesa Gallium driver 24.2.0 for AMD Radeon Pro VII (radeonsi, vega20, LLVM 18.1.8, DRM 3.57, 6.10.6-gentoo-dist).
Info: Driver not found in known nonstandard list, using standard behaviour.
Info: Input surface format is nv12.
Info: Using VAAPI profile VAProfileH264ConstrainedBaseline (13).
Info: Using VAAPI entrypoint VAEntrypointEncSlice (6).
Info: Using VAAPI render target format YUV420 (0x1).
Info: RC mode: CQP.
Info: RC quality: 5.
Info: RC framerate: 65535/1 (65535.00 fps).
Info: Driver does not report any additional prediction constraints.
Info: Using intra and P-frames (supported references: 1 / 0).
Warning: libav: 0: Driver does not support some wanted packed headers (wanted 0xd, found 0x1).
Info: Using level 4.
-----> Info: Choosing open-h264 encoding for client 0x55aa74ef3bb0
Info: Opened DRM device /dev/dri/renderD128: driver amdgpu version 3.57.0.
Info: w:1 h:1 pixfmt:drm_prime tb:1/1 fr:0/1 sar:0/1
Info: libva: VA-API version 1.22.0
Info: libva: Trying to open /usr/lib64/va/drivers/radeonsi_drv_video.so
Info: libva: Found init function __vaDriverInit_1_22
Info: libva: va_openDriver() returns 0
Info: Initialised VAAPI connection: version 1.22
Info: VAAPI driver: Mesa Gallium driver 24.2.0 for AMD Radeon Pro VII (radeonsi, vega20, LLVM 18.1.8, DRM 3.57, 6.10.6-gentoo-dist).