Segementation fault when using anything else but yolov7-320 TensorRT

[nordic-style]

Whenever i try to use anything else but yolov7-320 i get a segementation error. I tried 14.0.1 and the current dev package (0.15.0-a2f42d5)
Hardware: CPU 5800X3D GPU: 4070ti 12 GB

yolov7x-640 (segmentation fault)

docker compose up
[+] Running 1/0
 ✔ Container frigate  Recreated                                                                                                                                                                                 0.0s
Attaching to frigate
frigate  | s6-rc: info: service s6rc-fdholder: starting
frigate  | s6-rc: info: service s6rc-oneshot-runner: starting
frigate  | s6-rc: info: service s6rc-oneshot-runner successfully started
frigate  | s6-rc: info: service fix-attrs: starting
frigate  | s6-rc: info: service s6rc-fdholder successfully started
frigate  | s6-rc: info: service fix-attrs successfully started
frigate  | s6-rc: info: service legacy-cont-init: starting
frigate  | s6-rc: info: service legacy-cont-init successfully started
frigate  | s6-rc: info: service trt-model-prepare: starting
frigate  | s6-rc: info: service log-prepare: starting
frigate  | s6-rc: info: service log-prepare successfully started
frigate  | s6-rc: info: service nginx-log: starting
frigate  | s6-rc: info: service go2rtc-log: starting
frigate  | s6-rc: info: service frigate-log: starting
frigate  | s6-rc: info: service certsync-log: starting
frigate  | s6-rc: info: service nginx-log successfully started
frigate  | s6-rc: info: service go2rtc-log successfully started
frigate  | s6-rc: info: service go2rtc: starting
frigate  | s6-rc: info: service frigate-log successfully started
frigate  | s6-rc: info: service certsync-log successfully started
frigate  | s6-rc: info: service go2rtc successfully started
frigate  | s6-rc: info: service go2rtc-healthcheck: starting
frigate  | s6-rc: info: service go2rtc-healthcheck successfully started
frigate  | No models to convert.
frigate  | s6-rc: info: service trt-model-prepare successfully started
frigate  | s6-rc: info: service frigate: starting
frigate  | s6-rc: info: service frigate successfully started
frigate  | s6-rc: info: service nginx: starting
frigate  | 2024-10-09 01:03:30.039628482  [INFO] Preparing new go2rtc config...
frigate  | 2024-10-09 01:03:30.041517977  [INFO] Preparing Frigate...
frigate  | 2024-10-09 01:03:30.042237384  [INFO] Starting NGINX...
frigate  | 2024-10-09 01:03:30.044769721  [INFO] Starting Frigate...
frigate  | 2024-10-09 01:03:30.047714276  [INFO] No TLS certificate found. Generating a self signed certificate...
frigate  | 2024-10-09 01:03:30.283224276  [INFO] Starting go2rtc...
frigate  | 2024-10-09 01:03:30.347952862  01:03:30.347 INF go2rtc platform=linux/amd64 revision=b2399f3 version=1.9.2
frigate  | 2024-10-09 01:03:30.347964404  01:03:30.347 INF config path=/dev/shm/go2rtc.yaml
frigate  | 2024-10-09 01:03:30.348274529  01:03:30.348 INF [rtsp] listen addr=:8554
frigate  | 2024-10-09 01:03:30.348313603  01:03:30.348 INF [api] listen addr=:1984
frigate  | 2024-10-09 01:03:30.348375770  01:03:30.348 INF [webrtc] listen addr=:8555/tcp
frigate  | s6-rc: info: service nginx successfully started
frigate  | s6-rc: info: service certsync: starting
frigate  | s6-rc: info: service certsync successfully started
frigate  | s6-rc: info: service legacy-services: starting
frigate  | 2024-10-09 01:03:30.577591225  [INFO] Starting certsync...
frigate  | s6-rc: info: service legacy-services successfully started
frigate  | 2024-10-09 01:03:30.624686093  127.0.0.1 - - [09/Oct/2024:01:03:30 +0000] "" 400 0 "-" "-" "-"
frigate  | 2024-10-09 01:03:31.614026750  [2024-10-09 01:03:31] frigate.util.config            INFO    : Checking if frigate config needs migration...
frigate  | 2024-10-09 01:03:31.627944998  [2024-10-09 01:03:31] frigate.util.config            INFO    : frigate config does not need migration...
frigate  | 2024-10-09 01:03:31.635882593  [2024-10-09 01:03:31] frigate.config.config          WARNING : Customizing more than a detector model path is unsupported.
frigate  | 2024-10-09 01:03:31.863534388  [2024-10-09 01:03:31] frigate.app                    INFO    : Starting Frigate (0.15.0-a2f42d5)
frigate  | 2024-10-09 01:03:31.867830964  [2024-10-09 01:03:31] peewee_migrate.logs            INFO    : Starting migrations
frigate  | 2024-10-09 01:03:31.867976057  [2024-10-09 01:03:31] peewee_migrate.logs            INFO    : There is nothing to migrate
frigate  | 2024-10-09 01:03:31.876002610  [2024-10-09 01:03:31] frigate.app                    INFO    : Recording process started: 386
frigate  | 2024-10-09 01:03:31.881406435  [2024-10-09 01:03:31] frigate.app                    INFO    : Review process started: 395
frigate  | 2024-10-09 01:03:31.882930230  [2024-10-09 01:03:31] frigate.app                    INFO    : go2rtc process pid: 108
frigate  | 2024-10-09 01:03:31.903299208  [2024-10-09 01:03:31] detector.tensorrt              INFO    : Starting detection process: 417
frigate  | 2024-10-09 01:03:31.905975216  [2024-10-09 01:03:31] frigate.app                    INFO    : Output process started: 419
frigate  | 2024-10-09 01:03:31.914695538  [2024-10-09 01:03:31] frigate.app                    INFO    : Camera processor started for flur_oben: 435
frigate  | 2024-10-09 01:03:31.924608479  [2024-10-09 01:03:31] frigate.app                    INFO    : Capture process started for flur_oben: 442
frigate  | 2024-10-09 01:03:31.973455734  [2024-10-09 01:03:31] frigate.api.fastapi_app        INFO    : Starting FastAPI app
frigate  | 2024-10-09 01:03:32.504195561  [2024-10-09 01:03:32] frigate.detectors.plugins.tensorrt INFO    : Loaded engine size: 138 MiB
frigate  | 2024-10-09 01:03:32.589473856  [2024-10-09 01:03:32] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +6, GPU +8, now: CPU 306, GPU 347 (MiB)
frigate  | 2024-10-09 01:03:32.593282932  [2024-10-09 01:03:32] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +2, GPU +10, now: CPU 308, GPU 357 (MiB)
frigate  | 2024-10-09 01:03:32.603389479  [2024-10-09 01:03:32] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +141, now: CPU 0, GPU 141 (MiB)
frigate  | 2024-10-09 01:03:32.609402323  [2024-10-09 01:03:32] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 170, GPU 355 (MiB)
frigate  | 2024-10-09 01:03:32.609413093  [2024-10-09 01:03:32] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 170, GPU 363 (MiB)
frigate  | 2024-10-09 01:03:32.609439312  [2024-10-09 01:03:32] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +75, now: CPU 0, GPU 216 (MiB)
frigate  | 2024-10-09 01:03:37.638703876  Fatal Python error: Segmentation fault
frigate  | 2024-10-09 01:03:37.638706150
frigate  | 2024-10-09 01:03:37.638706852  Thread 0x00007186d34006c0 (most recent call first):
frigate  | 2024-10-09 01:03:37.638719756    File "/usr/lib/python3.9/threading.py", line 312 in wait
frigate  | 2024-10-09 01:03:37.638743701    File "/usr/lib/python3.9/multiprocessing/queues.py", line 233 in _feed
frigate  | 2024-10-09 01:03:37.638761685    File "/usr/lib/python3.9/threading.py", line 892 in run
frigate  | 2024-10-09 01:03:37.638791181    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
frigate  | 2024-10-09 01:03:37.638813673    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
frigate  | 2024-10-09 01:03:37.638814054
frigate  | 2024-10-09 01:03:37.638818613  Current thread 0x00007187bfa5d740 (most recent call first):
frigate  | 2024-10-09 01:03:37.638845553    File "/opt/frigate/frigate/detectors/plugins/tensorrt.py", line 193 in <listcomp>
frigate  | 2024-10-09 01:03:37.638877393    File "/opt/frigate/frigate/detectors/plugins/tensorrt.py", line 192 in _do_inference
frigate  | 2024-10-09 01:03:37.638906629    File "/opt/frigate/frigate/detectors/plugins/tensorrt.py", line 309 in detect_raw
frigate  | 2024-10-09 01:03:37.638930173    File "/opt/frigate/frigate/object_detection.py", line 83 in detect_raw
frigate  | 2024-10-09 01:03:37.638953778    File "/opt/frigate/frigate/object_detection.py", line 133 in run_detector
frigate  | 2024-10-09 01:03:37.638978885    File "/usr/lib/python3.9/multiprocessing/process.py", line 108 in run
frigate  | 2024-10-09 01:03:37.639002610    File "/opt/frigate/frigate/util/process.py", line 41 in run_wrapper
frigate  | 2024-10-09 01:03:37.639028088    File "/usr/lib/python3.9/multiprocessing/process.py", line 315 in _bootstrap
frigate  | 2024-10-09 01:03:37.639058565    File "/usr/lib/python3.9/multiprocessing/popen_fork.py", line 71 in _launch
frigate  | 2024-10-09 01:03:37.639088973    File "/usr/lib/python3.9/multiprocessing/popen_fork.py", line 19 in __init__
frigate  | 2024-10-09 01:03:37.639121785    File "/usr/lib/python3.9/multiprocessing/context.py", line 277 in _Popen
frigate  | 2024-10-09 01:03:37.639145579    File "/usr/lib/python3.9/multiprocessing/context.py", line 224 in _Popen
frigate  | 2024-10-09 01:03:37.639169224    File "/usr/lib/python3.9/multiprocessing/process.py", line 121 in start
frigate  | 2024-10-09 01:03:37.639190424    File "/opt/frigate/frigate/util/process.py", line 30 in start
frigate  | 2024-10-09 01:03:37.639217706    File "/opt/frigate/frigate/object_detection.py", line 192 in start_or_restart
frigate  | 2024-10-09 01:03:37.639240529    File "/opt/frigate/frigate/object_detection.py", line 160 in __init__
frigate  | 2024-10-09 01:03:37.639261378    File "/opt/frigate/frigate/app.py", line 347 in start_detectors
frigate  | 2024-10-09 01:03:37.639279552    File "/opt/frigate/frigate/app.py", line 588 in start
frigate  | 2024-10-09 01:03:37.639309539    File "/opt/frigate/frigate/__main__.py", line 60 in main
frigate  | 2024-10-09 01:03:37.639334336    File "/opt/frigate/frigate/__main__.py", line 64 in <module>
frigate  | 2024-10-09 01:03:37.639362779    File "/usr/lib/python3.9/runpy.py", line 87 in _run_code
frigate  | 2024-10-09 01:03:37.639395741    File "/usr/lib/python3.9/runpy.py", line 197 in _run_module_as_main
frigate  | 2024-10-09 01:03:40.040200819  [INFO] Starting go2rtc healthcheck service...`

yolov7-320 (working):

docker compose up
[+] Running 1/0
 ✔ Container frigate  Created                                                                                                                                                                                                                                                                                                                                                                                                          0.0s
Attaching to frigate
frigate  | s6-rc: info: service s6rc-fdholder: starting
frigate  | s6-rc: info: service s6rc-oneshot-runner: starting
frigate  | s6-rc: info: service s6rc-oneshot-runner successfully started
frigate  | s6-rc: info: service fix-attrs: starting
frigate  | s6-rc: info: service s6rc-fdholder successfully started
frigate  | s6-rc: info: service fix-attrs successfully started
frigate  | s6-rc: info: service legacy-cont-init: starting
frigate  | s6-rc: info: service legacy-cont-init successfully started
frigate  | s6-rc: info: service trt-model-prepare: starting
frigate  | s6-rc: info: service log-prepare: starting
frigate  | s6-rc: info: service log-prepare successfully started
frigate  | s6-rc: info: service nginx-log: starting
frigate  | s6-rc: info: service go2rtc-log: starting
frigate  | s6-rc: info: service frigate-log: starting
frigate  | s6-rc: info: service certsync-log: starting
frigate  | s6-rc: info: service nginx-log successfully started
frigate  | s6-rc: info: service go2rtc-log successfully started
frigate  | s6-rc: info: service go2rtc: starting
frigate  | s6-rc: info: service frigate-log successfully started
frigate  | s6-rc: info: service certsync-log successfully started
frigate  | s6-rc: info: service go2rtc successfully started
frigate  | s6-rc: info: service go2rtc-healthcheck: starting
frigate  | s6-rc: info: service go2rtc-healthcheck successfully started
frigate  | No models to convert.
frigate  | s6-rc: info: service trt-model-prepare successfully started
frigate  | s6-rc: info: service frigate: starting
frigate  | s6-rc: info: service frigate successfully started
frigate  | s6-rc: info: service nginx: starting
frigate  | 2024-10-09 01:05:56.010426477  [INFO] Preparing new go2rtc config...
frigate  | 2024-10-09 01:05:56.012324227  [INFO] Preparing Frigate...
frigate  | 2024-10-09 01:05:56.013010973  [INFO] Starting NGINX...
frigate  | 2024-10-09 01:05:56.015638871  [INFO] Starting Frigate...
frigate  | s6-rc: info: service nginx successfully started
frigate  | s6-rc: info: service certsync: starting
frigate  | s6-rc: info: service certsync successfully started
frigate  | s6-rc: info: service legacy-services: starting
frigate  | 2024-10-09 01:05:56.080250571  [INFO] Starting certsync...
frigate  | s6-rc: info: service legacy-services successfully started
frigate  | 2024-10-09 01:05:56.176057687  127.0.0.1 - - [09/Oct/2024:01:05:56 +0000] "" 400 0 "-" "-" "-"
frigate  | 2024-10-09 01:05:56.253065795  [INFO] Starting go2rtc...
frigate  | 2024-10-09 01:05:56.317682114  01:05:56.317 INF go2rtc platform=linux/amd64 revision=b2399f3 version=1.9.2
frigate  | 2024-10-09 01:05:56.317690860  01:05:56.317 INF config path=/dev/shm/go2rtc.yaml
frigate  | 2024-10-09 01:05:56.318031443  01:05:56.317 INF [api] listen addr=:1984
frigate  | 2024-10-09 01:05:56.318074183  01:05:56.318 INF [rtsp] listen addr=:8554
frigate  | 2024-10-09 01:05:56.318260896  01:05:56.318 INF [webrtc] listen addr=:8555/tcp
frigate  | 2024-10-09 01:05:57.524141872  [2024-10-09 01:05:57] frigate.util.config            INFO    : Checking if frigate config needs migration...
frigate  | 2024-10-09 01:05:57.538149149  [2024-10-09 01:05:57] frigate.util.config            INFO    : frigate config does not need migration...
frigate  | 2024-10-09 01:05:57.546111221  [2024-10-09 01:05:57] frigate.config.config          WARNING : Customizing more than a detector model path is unsupported.
frigate  | 2024-10-09 01:05:57.667955940  [2024-10-09 01:05:57] frigate.app                    INFO    : Starting Frigate (0.15.0-a2f42d5)
frigate  | 2024-10-09 01:05:57.672012824  [2024-10-09 01:05:57] peewee_migrate.logs            INFO    : Starting migrations
frigate  | 2024-10-09 01:05:57.672149282  [2024-10-09 01:05:57] peewee_migrate.logs            INFO    : There is nothing to migrate
frigate  | 2024-10-09 01:05:57.684279547  [2024-10-09 01:05:57] frigate.app                    INFO    : Recording process started: 384
frigate  | 2024-10-09 01:05:57.684483562  [2024-10-09 01:05:57] frigate.app                    INFO    : Review process started: 393
frigate  | 2024-10-09 01:05:57.686047162  [2024-10-09 01:05:57] frigate.app                    INFO    : go2rtc process pid: 109
frigate  | 2024-10-09 01:05:57.706218769  [2024-10-09 01:05:57] detector.tensorrt              INFO    : Starting detection process: 415
frigate  | 2024-10-09 01:05:57.708966724  [2024-10-09 01:05:57] frigate.app                    INFO    : Output process started: 417
frigate  | 2024-10-09 01:05:57.717366661  [2024-10-09 01:05:57] frigate.app                    INFO    : Camera processor started for flur_oben: 433
frigate  | 2024-10-09 01:05:57.723586676  [2024-10-09 01:05:57] frigate.app                    INFO    : Capture process started for flur_oben: 440
frigate  | 2024-10-09 01:05:57.776895101  [2024-10-09 01:05:57] frigate.api.fastapi_app        INFO    : Starting FastAPI app
frigate  | 2024-10-09 01:05:58.273964041  [2024-10-09 01:05:58] frigate.detectors.plugins.tensorrt INFO    : Loaded engine size: 71 MiB
frigate  | 2024-10-09 01:05:58.352703085  [2024-10-09 01:05:58] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +6, GPU +8, now: CPU 236, GPU 281 (MiB)
frigate  | 2024-10-09 01:05:58.356436669  [2024-10-09 01:05:58] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +2, GPU +10, now: CPU 238, GPU 291 (MiB)
frigate  | 2024-10-09 01:05:58.365612240  [2024-10-09 01:05:58] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +71, now: CPU 0, GPU 71 (MiB)
frigate  | 2024-10-09 01:05:58.365625415  [2024-10-09 01:05:58] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 167, GPU 283 (MiB)
frigate  | 2024-10-09 01:05:58.365647627  [2024-10-09 01:05:58] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 167, GPU 291 (MiB)
frigate  | 2024-10-09 01:05:58.365666553  [2024-10-09 01:05:58] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +14, now: CPU 0, GPU 85 (MiB)
frigate  | 2024-10-09 01:06:06.011014994  [INFO] Starting go2rtc healthcheck service...`

docker-compose.yml

services:
  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:a2f42d5-tensorrt
#    image: ghcr.io/blakeblackshear/frigate:0.14.1-tensorrt
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1 # number of GPUs
              capabilities: [gpu]
    shm_size: "1024mb" # for one FHD cam
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /data/frigate/config:/config
      - /data/frigate/storage:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "5000:5000"
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
    environment:
      - FRIGATE_RTSP_PASSWORD=xxx
      - YOLO_MODELS=yolov4-416,yolov4-608,yolov7-320,yolov7x-640
      - USE_FP16=true
    runtime: nvidia

config.yml

mqtt:
  host: 192.168.40.11
  user: mqtt-frigate
  password: XXX

ffmpeg:
  hwaccel_args: preset-nvidia

detectors:
  tensorrt:
    type: tensorrt
    device: 0
    model:
      path: /config/model_cache/tensorrt/yolov7x-640.trt
#      path: /config/model_cache/tensorrt/yolov4-608.trt
#      path: /config/model_cache/tensorrt/yolov7-320.trt
      input_tensor: nchw
      input_pixel_format: rgb
      width: 640
      height: 640

record:
  enabled: true
  retain:
    days: 7
    mode: motion
  alerts:
    retain:
      days: 30
  detections:
    retain:
      days: 30
snapshots:
  enabled: true
  retain:
    default: 30

cameras:
  flur_oben:
    detect:
      width: 1920
      height: 1080
      fps: 5
    ffmpeg:
      inputs:
        - path:
            rtsp://root:[email protected]/axis-media/media.amp?adjustablelivestream=1&camera=5
          roles:
            - detect
    objects:
      track:
        - person
        - dog
      mask:
        0.192,0.161,0.245,0.512,0.335,0.745,0.513,0.873,0.671,0.824,0.746,0.658,0.796,0.428,0.859,0.244,0.618,0.084,0.379,0.116
    motion: {}
    zones:
      Eingang:
        coordinates: 0.169,0.176,0.397,0.975,0.715,0.879,0.873,0.238,0.617,0.096
        loitering_time: 0
    review:
      alerts:
        required_zones: Eingang
version: 0.15-0

Any idea what i am doing wrong? Or is that a memory calculation bug?

[nordic-style]

I have restarted everything and tried other driver configurations from this thread (and all other options) #9801 - to no success. I've also tried a different GPU (ada 6000) - also to no success. I'm currently trying to create a debug environment for the detector to debug stuff - web frontend was easy - if anyone has a tip on how to run the detector part (for debugging) I'd be very grateful.

[NickM-27]

the dev container should work with tensorrt, docker compose just needs to be edited to have some stuff uncommented