Missing option to set h264_nvenc as default encoder

[vladimir-hidalgo-dev]

I think there's missing .env option to set h264_nvenc as default encoder for those who want to use nvidia or other GPUs.

You can't set it with the current ENV options

[saghul]

In what context? Jibri perhaps?

[vladimir-hidalgo-dev]

Correct, sorry that initial ticket was sent at the end of a 10h attempt to get it working.

What I'm trying to do is to change -c:v libx264 to -c:v h264_nvenc when running ffmpeg:

jibri-7 | Jibri 2025-02-15 08:15:21.880 INFO: [191] [session_id=c54d1a86-92f5-43d9-8faf-cfe4f1358bd0] JibriSubprocess.launch#42: Starting ffmpeg with command ffmpeg -y -v info -f x11grab -draw_mouse 0 -r 30 -s 1280x720 -thread_queue_size 4096 -i :0.0+0,0 -f pulse -thread_queue_size 4096 -i default -acodec aac -strict -2 -ar 44100 -b:a 128k -af aresample=async=1 -c:v libx264 -preset veryfast -profile:v main -level 3.1 -pix_fmt yuv420p -r 30 -crf 25 -g 60 -tune zerolatency -timestamp now -f mp4 /config/recordings/c54d1a86-92f5-43d9-8faf-cfe4f1358bd0/t42990_2025-02-15-08-15-18.mp4 ([ffmpeg, -y, -v, info, -f, x11grab, -draw_mouse, 0, -r, 30, -s, 1280x720, -thread_queue_size, 4096, -i, :0.0+0,0, -f, pulse, -thread_queue_size, 4096, -i, default, -acodec, aac, -strict, -2, -ar, 44100, -b:a, 128k, -af, aresample=async=1, -c:v, libx264, -preset, veryfast, -profile:v, main, -level, 3.1, -pix_fmt, yuv420p, -r, 30, -crf, 25, -g, 60, -tune, zerolatency, -timestamp, now, -f, mp4, /config/recordings/c54d1a86-92f5-43d9-8faf-cfe4f1358bd0/t42990_2025-02-15-08-15-18.mp4])

I see the conf value here:
https://github.com/jitsi/jibri/blob/master/src/main/resources/reference.conf#L103

and here the override layer in Docker:
https://github.com/jitsi/docker-jitsi-meet/blob/master/jibri/rootfs/defaults/jibri.conf#L44

However I can't find a way to do this change as it doesn't seem to be tied to a ENV.

I saw this include "custom-jibri.conf"
https://github.com/jitsi/docker-jitsi-meet/blob/master/jibri/rootfs/defaults/jibri.conf#L93

And tried

volumes:
            - ${CONFIG}/jibri:/config:Z
            - ${CONFIG}/jibri/custom-jibri.conf:/etc/jitsi/jibri/custom-jibri.conf

with custom-jibri.conf being

video-params = [
      "-c:v", "h264_nvenc"
      "-pix_fmt", "yuv420p",
      "-r", ${jibri.ffmpeg.framerate}
      "-crf", ${jibri.ffmpeg.h264-constant-rate-factor}
]

But it had no effect, so I thought filing an issue for this was the best option at this point.

Thank you @saghul

[vladimir-hidalgo-dev]

I've also attempted set custom-jibri.conf like this

jibri {
        ffmpeg {
                video-params = [
                      "-c:v", "h264_nvenc"
                      "-pix_fmt", "yuv420p",
                      "-r", ${jibri.ffmpeg.framerate}
                      "-crf", ${jibri.ffmpeg.h264-constant-rate-factor}
                ]
        }
}

[vladimir-hidalgo-dev]

And also custom-jibri.conf like this:

jibri.ffmpeg.video-params = [
      "-c:v", "h264_nvenc"
      "-pix_fmt", "yuv420p",
      "-r", ${jibri.ffmpeg.framerate}
      "-crf", ${jibri.ffmpeg.h264-constant-rate-factor}
]

[saghul]

@bgrozev Ping :-) Should this work after your recent changes in Jibri?

[vladimir-hidalgo-dev]

I forgot to mention that I tested this with version 10008

[bgrozev]

This should work since jibri 8.0-176-gac5ccc9 (this was merged very recently). I don't know what jibri version comes with 10008.

[bgrozev]

Note that h264_nvenc might require different flags to be configured. If you get to a set of options that work well we'd appreciate an update, so we can put it in a doc somewhere. :)

[vladimir-hidalgo-dev]

Thank you! and it worked for me like this:

ffmpeg-wrapper.sh because I couldn't set the options

#!/bin/bash

FILTERED_ARGS=$(echo "$@" | sed 's/libx264/h264_nvenc/g')
FILTERED_ARGS=$(echo "$FILTERED_ARGS" | sed 's/-preset [^ ]*//g')
FILTERED_ARGS=$(echo "$FILTERED_ARGS" | sed 's/-tune [^ ]*//g')

FILTERED_ARGS=" -hwaccel cuda -hwaccel_output_format cuda $FILTERED_ARGS"

# Echo the modified command
echo "Executing FFmpeg with modified arguments:"
echo "/usr/bin/ffmpeg $FILTERED_ARGS"

# Run FFmpeg with modified arguments
exec /usr/bin/ffmpeg $FILTERED_ARGS

docker-compose.yml

services:
    jibri:
        image: jitsi/jibri:${JITSI_IMAGE_VERSION:-stable-10008}
        restart: ${RESTART_POLICY:-unless-stopped}
        volumes:
            - ${CONFIG}/jibri:/config:Z
            - ${CONFIG}/jibri/ffmpeg-wrapper.sh:/usr/local/bin/ffmpeg
        shm_size: '3gb'
        cap_add:
            - SYS_ADMIN
        runtime: nvidia
        environment:
            - NVIDIA_VISIBLE_DEVICES=all
            - NVIDIA_DRIVER_CAPABILITIES=all
            - ....
        deploy:
            replicas: 20

works perfectly fine and reduces the CPU a lot! I'm now looking at getting Chrome to use the GPU as well.

[saghul]

works perfectly fine and reduces the CPU a lot! I'm now looking at getting Chrome to use the GPU as well.

That's nice to hear! In terms of resources and cost, do you find it's worth it? IIRC we went with CPU back in the day because instances with access to a GPU are significantly more expensive so it was not worth it.

[vladimir-hidalgo-dev]

It's actually cheaper!

n1-standard-32 (32 vCPU, 16 core, 120 GB memory, 1 NVIDIA T4)
$759.11 - good for 20 Jibri concurrent recordings

c2d-highcpu-32 (32 vCPU, 16 core, 64 GB memory)
$876.56 -good for 12 Jibri concurrent recordings

c4-highcpu-32 (32 vCPU, 16 core, 64 GB memory)
$995.11 - good for 10 Jibri concurrent recordings

---

That's why I'm looking moving Chrome to the GPU as well, right now 20 Jibri recording sessions use only 12% of the GPU

[saghul]

Interesting! Can you run multiple jibris on that instance using the same GPU simultaneously?

[saghul]

Ping @aaronkvanmeerten

[vladimir-hidalgo-dev]

@saghul yup! you can totally have multiple instances using the same GPU simultaneously.

I'm trying to get Chrome in there because it's still underused even at 20 Jibri's ffmpeg instances

[vladimir-hidalgo-dev]

Finally ChromeDriver rendering via GPU, 20 Jibri recordings at 1080p, I think I could push it now to 30.

[saghul]

What did you do to make it use hw acceleration?