[Detector Support]: Error creating tensorrt model

[tichachm]

Describe the problem you are having

Hi, I am running frigate in LXC Container on Proxmox with a Nvidia Quadro P600.
It was working fine unitl I updated Proxmox and rebooted.
After that there was a permission error at container start, so I had to update the nvidia drivers on host and in LXC to fix that.

But now I can no longer start frigate. Tensorrt model seems to be regenearted but this does not work.

Here ist he Log output:

Generating the following TRT Models: yolov7-320
Downloading yolo weights
s6-rc: info: service log-prepare successfully started
s6-rc: info: service nginx-log: starting
s6-rc: info: service go2rtc-log: starting
s6-rc: info: service frigate-log: starting
s6-rc: info: service nginx-log successfully started
s6-rc: info: service frigate-log successfully started
s6-rc: info: service go2rtc-log successfully started
s6-rc: info: service go2rtc: starting
s6-rc: info: service go2rtc successfully started
s6-rc: info: service go2rtc-healthcheck: starting
Creating yolov7-320.cfg and yolov7-320.weights
s6-rc: info: service go2rtc-healthcheck successfully started
Done.
2024-03-30 10:35:10.106185356  [INFO] Preparing new go2rtc config...
2024-03-30 10:35:10.435004763  [INFO] Starting go2rtc...
2024-03-30 10:35:10.529688605  10:35:10.528 INF go2rtc version 1.8.4 linux/amd64
2024-03-30 10:35:10.529692984  10:35:10.528 INF [rtsp] listen addr=:8554
2024-03-30 10:35:10.529697582  10:35:10.529 INF [webrtc] listen addr=:8555
2024-03-30 10:35:10.530035045  10:35:10.529 INF [api] listen addr=:1984
Generating yolov7-320.trt. This may take a few minutes.
Traceback (most recent call last):
  File "/usr/local/src/tensorrt_demos/yolo/onnx_to_tensorrt.py", line 214, in <module>
    main()
  File "/usr/local/src/tensorrt_demos/yolo/onnx_to_tensorrt.py", line 202, in main
    engine = build_engine(
  File "/usr/local/src/tensorrt_demos/yolo/onnx_to_tensorrt.py", line 112, in build_engine
    with trt.Builder(TRT_LOGGER) as builder, builder.create_network(*EXPLICIT_BATCH) as network, trt.OnnxParser(network, TRT_LOGGER) as parser:
TypeError: pybind11::init(): factory function returned nullptr
[03/30/2024-10:35:17] [TRT] [W] Unable to determine GPU memory usage
[03/30/2024-10:35:17] [TRT] [W] Unable to determine GPU memory usage
[03/30/2024-10:35:17] [TRT] [W] CUDA initialization failure with error: 999. Please check your CUDA installation:  http://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html
Loading the ONNX file...
Available tensorrt models:
ls: cannot access '*.trt': No such file or directory
s6-rc: warning: unable to start service trt-model-prepare: command exited 2
2024-03-30 10:35:20.083513669  [INFO] Starting go2rtc healthcheck service...

Nvidia drivers seems to be ok:

docker run -it --rm --gpus all ubuntu nvidia-smi
Sat Mar 30 09:42:33 2024       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.146.02             Driver Version: 535.146.02   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Quadro P600                    Off | 00000000:01:00.0 Off |                  N/A |
| 22%   33C    P0              N/A /  N/A |      0MiB /  2048MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

What does CUDA initialization failure with error: 999 mean?
Thanks

Version

0.13.2

Frigate config file

database:
  path: /db/frigate.db
ffmpeg:
  hwaccel_args: preset-nvidia-h264
detectors:
  tensorrt:
    type: tensorrt
    device: 0 #This is the default, select the first GPU
#0.13
model:
  path: /config/model_cache/tensorrt/yolov7-320.trt
  input_tensor: nchw
  input_pixel_format: rgb
  width: 320
  height: 320
birdseye:
  enabled: true
  mode: motion
# Optional: Object configuration
# NOTE: Can be overridden at the camera level
objects:
  # Optional: list of objects to track from labelmap.txt (default: shown below)
  track:
  - person
  - cat
  - dog
  - bird
  - mouse
  - sheep
  - horse
  - cow

docker-compose file or Docker CLI command

services:
  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:stable-tensorrt
#    image: ghcr.io/blakeblackshear/frigate:0.13.0-tensorrt
    shm_size: "200mb" # update for your cameras based on calculation above
    deploy:    # <------------- Add this section
      resources:
        reservations:
          devices:
            - driver: nvidia
#              device_ids: ['0'] # this is only needed when using multiple GPUs
              count: all # number of GPUs
              capabilities: [gpu]
    devices:
      - /dev/bus/usb:/dev/bus/usb # passes the USB Coral, needs to be modified for other versions
      #- /dev/apex_0:/dev/apex_0 # passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-on-linux
      - /dev/dri/renderD128 # for intel hwaccel, needs to be updated for your hardware
    volumes:
#      - ./trt-models://trt-models
      - /etc/localtime:/etc/localtime:ro
      - ./config:/config
      - ./db:/db
      - /dataPool/daten/ueberwachung/frigate:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "5000:5000"
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
#    runtime: nvidia
    environment:
      NVIDIA_VISIBLE_DEVICES: "all"
      NVIDIA_DRIVER_CAPABILITIES: "compute,utility,video"
      USE_FP16: "False"
#      YOLO_MODELS: "yolov4-288"
    privileged: true

Relevant log output

2024-03-30 10:35:10.106185356  [INFO] Preparing new go2rtc config...
2024-03-30 10:35:10.435004763  [INFO] Starting go2rtc...
2024-03-30 10:35:10.529688605  10:35:10.528 INF go2rtc version 1.8.4 linux/amd64
2024-03-30 10:35:10.529692984  10:35:10.528 INF [rtsp] listen addr=:8554
2024-03-30 10:35:10.529697582  10:35:10.529 INF [webrtc] listen addr=:8555
2024-03-30 10:35:10.530035045  10:35:10.529 INF [api] listen addr=:1984
Generating yolov7-320.trt. This may take a few minutes.
Traceback (most recent call last):
  File "/usr/local/src/tensorrt_demos/yolo/onnx_to_tensorrt.py", line 214, in <module>
    main()
  File "/usr/local/src/tensorrt_demos/yolo/onnx_to_tensorrt.py", line 202, in main
    engine = build_engine(
  File "/usr/local/src/tensorrt_demos/yolo/onnx_to_tensorrt.py", line 112, in build_engine
    with trt.Builder(TRT_LOGGER) as builder, builder.create_network(*EXPLICIT_BATCH) as network, trt.OnnxParser(network, TRT_LOGGER) as parser:
TypeError: pybind11::init(): factory function returned nullptr
[03/30/2024-10:35:17] [TRT] [W] Unable to determine GPU memory usage
[03/30/2024-10:35:17] [TRT] [W] Unable to determine GPU memory usage
[03/30/2024-10:35:17] [TRT] [W] CUDA initialization failure with error: 999. Please check your CUDA installation:  http://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html
Loading the ONNX file...
Available tensorrt models:
ls: cannot access '*.trt': No such file or directory
s6-rc: warning: unable to start service trt-model-prepare: command exited 2
2024-03-30 10:35:20.083513669  [INFO] Starting go2rtc healthcheck service...

Operating system

Proxmox

Install method

Docker Compose

Coral version

Other

Any other information that may be helpful

No response

[NickM-27]

This might be fixed by #10210

[tichachm]

Hi,
thanks for your response. I could fix the problem by reinstalling nvidia drivers on Host and in the LXC container. After that I had to update lxc.cgroup values in the conntainer config. Now it is working again.
Thanks