[Detector Support]: 4070 ti detection does not work (setup does) with yolov7-320 and crashes with bigger models. #14465

nordic-style · 2024-10-21T00:38:07Z

nordic-style
Oct 21, 2024

Describe the problem you are having

As soon as i switch from cpu to tensorRT the detector does not detect anything. As soon as i try to load a model that is larger than 320x320 frigate crashes with a segmentation fault. I am using the latest driver from the nvidia website and nvidia-smi tells me that the process is running fine (with 320x320 models) together with the decoding process. I also tried 550 and 555 drivers without success.

Version

0.14.1-f4f3cfa

Frigate config file

logger:
  default: debug

mqtt:
  host: 192.168.40.11
  user: mqtt-frigate
  password: asdf

ffmpeg:
  hwaccel_args: preset-nvidia-h264

detectors:
  tensorrt:
#    type: cpu
    type: tensorrt
    model:
#      path: /config/model_cache/tensorrt/yolov4-608.trt
      path: /config/model_cache/tensorrt/yolov7-320.trt
#      input_tensor: nchw
#      input_pixel_format: rgb
#      width: 320
#      height: 320

record:
  enabled: true
  retain:
    days: 7
    mode: motion
snapshots:
  enabled: true
  retain:
    default: 30

cameras:
  test:
    detect:
      width: 1280
      height: 720
      fps: 6
      enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://admin:[email protected]/Preview_01_main
          roles:
            - detect
version: 0.14

docker-compose file or Docker CLI command

services:
  frigate:
    container_name: frigate
    privileged: true
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:0.14.1-tensorrt
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1 # number of GPUs
              capabilities: [gpu]
    shm_size: "2048mb" # for one FHD cam
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /data/frigate/config:/config
      - /data/frigate/storage:/media/frigate
      - type: tmpfs
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "5000:5000"
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
    environment:
      - FRIGATE_RTSP_PASSWORD=asdf
#      - YOLO_MODELS=yolov4-608
      - YOLO_MODELS=yolov7-320
      - USE_FP16=true
    runtime: nvidia

Relevant Frigate log output

no detection with yolov7-320:
frigate  | s6-rc: info: service s6rc-fdholder: starting
frigate  | s6-rc: info: service s6rc-oneshot-runner: starting
frigate  | s6-rc: info: service s6rc-oneshot-runner successfully started
frigate  | s6-rc: info: service fix-attrs: starting
frigate  | s6-rc: info: service s6rc-fdholder successfully started
frigate  | s6-rc: info: service fix-attrs successfully started
frigate  | s6-rc: info: service legacy-cont-init: starting
frigate  | s6-rc: info: service legacy-cont-init successfully started
frigate  | s6-rc: info: service trt-model-prepare: starting
frigate  | s6-rc: info: service log-prepare: starting
frigate  | No models to convert.
frigate  | s6-rc: info: service trt-model-prepare successfully started
frigate  | s6-rc: info: service log-prepare successfully started
frigate  | s6-rc: info: service nginx-log: starting
frigate  | s6-rc: info: service go2rtc-log: starting
frigate  | s6-rc: info: service frigate-log: starting
frigate  | s6-rc: info: service certsync-log: starting
frigate  | s6-rc: info: service go2rtc-log successfully started
frigate  | s6-rc: info: service go2rtc: starting
frigate  | s6-rc: info: service nginx-log successfully started
frigate  | s6-rc: info: service frigate-log successfully started
frigate  | s6-rc: info: service certsync-log successfully started
frigate  | s6-rc: info: service go2rtc successfully started
frigate  | s6-rc: info: service go2rtc-healthcheck: starting
frigate  | s6-rc: info: service frigate: starting
frigate  | s6-rc: info: service go2rtc-healthcheck successfully started
frigate  | s6-rc: info: service frigate successfully started
frigate  | s6-rc: info: service nginx: starting
frigate  | 2024-10-21 00:28:11.476174196  [INFO] Preparing Frigate...
frigate  | 2024-10-21 00:28:11.477157453  [INFO] Starting NGINX...
frigate  | 2024-10-21 00:28:11.478560301  [INFO] Preparing new go2rtc config...
frigate  | 2024-10-21 00:28:11.481859709  [INFO] No TLS certificate found. Generating a self signed certificate...
frigate  | 2024-10-21 00:28:11.482798502  [INFO] Starting Frigate...
frigate  | s6-rc: info: service nginx successfully started
frigate  | s6-rc: info: service certsync: starting
frigate  | s6-rc: info: service certsync successfully started
frigate  | s6-rc: info: service legacy-services: starting
frigate  | 2024-10-21 00:28:11.637710872  [INFO] Starting certsync...
frigate  | s6-rc: info: service legacy-services successfully started
frigate  | 2024-10-21 00:28:11.676907005  127.0.0.1 - - [21/Oct/2024:00:28:11 +0000] "" 400 0 "-" "-" "-"
frigate  | 2024-10-21 00:28:11.709319052  [INFO] Starting go2rtc...
frigate  | 2024-10-21 00:28:11.792894253  00:28:11.792 INF go2rtc platform=linux/amd64 revision=b2399f3 version=1.9.2
frigate  | 2024-10-21 00:28:11.792896638  00:28:11.792 INF config path=/dev/shm/go2rtc.yaml
frigate  | 2024-10-21 00:28:11.793189270  00:28:11.793 INF [rtsp] listen addr=:8554
frigate  | 2024-10-21 00:28:11.793210781  00:28:11.793 INF [api] listen addr=:1984
frigate  | 2024-10-21 00:28:11.793296433  00:28:11.793 INF [webrtc] listen addr=:8555/tcp
frigate  | 2024-10-21 00:28:12.444574040  [2024-10-21 00:28:12] frigate.app                    INFO    : Starting Frigate (0.14.1-f4f3cfa)
frigate  | 2024-10-21 00:28:12.444606631  [2024-10-21 00:28:12] frigate.util.config            INFO    : Checking if frigate config needs migration...
frigate  | 2024-10-21 00:28:12.452658401  [2024-10-21 00:28:12] frigate.util.config            INFO    : frigate config does not need migration...
frigate  | 2024-10-21 00:28:12.462199142  [2024-10-21 00:28:12] frigate.config                 WARNING : Customizing more than a detector model path is unsupported.
frigate  | 2024-10-21 00:28:12.560191299  [2024-10-21 00:28:12] peewee                         DEBUG   : ('CREATE TABLE IF NOT EXISTS "migratehistory" ("id" INTEGER NOT NULL PRIMARY KEY, "name" VARCHAR(255) NOT NULL, "migrated_at" DATETIME NOT NULL)', [])
frigate  | 2024-10-21 00:28:12.561261640  [2024-10-21 00:28:12] peewee                         DEBUG   : ('SELECT "t1"."id", "t1"."name", "t1"."migrated_at" FROM "migratehistory" AS "t1" ORDER BY "t1"."id"', [])
frigate  | 2024-10-21 00:28:12.562490199  [2024-10-21 00:28:12] peewee_migrate.logs            INFO    : Starting migrations
frigate  | 2024-10-21 00:28:12.562519575  [2024-10-21 00:28:12] peewee                         DEBUG   : ('SELECT "t1"."id", "t1"."name", "t1"."migrated_at" FROM "migratehistory" AS "t1" ORDER BY "t1"."id"', [])
frigate  | 2024-10-21 00:28:12.562976998  [2024-10-21 00:28:12] peewee_migrate.logs            INFO    : There is nothing to migrate
frigate  | 2024-10-21 00:28:12.571531527  [2024-10-21 00:28:12] frigate.app                    INFO    : Recording process started: 382
frigate  | 2024-10-21 00:28:12.575916956  [2024-10-21 00:28:12] asyncio                        DEBUG   : Using selector: EpollSelector
frigate  | 2024-10-21 00:28:12.576494877  [2024-10-21 00:28:12] frigate.app                    INFO    : Recording process started: 391
frigate  | 2024-10-21 00:28:12.578002263  [2024-10-21 00:28:12] frigate.app                    INFO    : go2rtc process pid: 105
frigate  | 2024-10-21 00:28:12.593662325  [2024-10-21 00:28:12] detector.tensorrt              INFO    : Starting detection process: 412
frigate  | 2024-10-21 00:28:12.596414601  [2024-10-21 00:28:12] frigate.app                    INFO    : Output process started: 414
frigate  | 2024-10-21 00:28:12.597955249  [2024-10-21 00:28:12] peewee.sqliteq                 DEBUG   : received query DELETE FROM "regions" WHERE NOT ("regions"."camera" IN (?))
frigate  | 2024-10-21 00:28:12.597994173  [2024-10-21 00:28:12] peewee                         DEBUG   : ('DELETE FROM "regions" WHERE NOT ("regions"."camera" IN (?))', ['test'])
frigate  | 2024-10-21 00:28:12.598500739  [2024-10-21 00:28:12] peewee                         DEBUG   : ('SELECT "t1"."camera", "t1"."grid", "t1"."last_update" FROM "regions" AS "t1" WHERE ("t1"."camera" = ?) LIMIT ? OFFSET ?', ['test', 1, 0])
frigate  | 2024-10-21 00:28:12.600469756  [2024-10-21 00:28:12] peewee                         DEBUG   : ('SELECT "t1"."id" FROM "event" AS "t1" WHERE ((("t1"."camera" = ?) AND (("t1"."false_positive" IS NULL) OR ("t1"."false_positive" = ?))) AND ("t1"."start_time" > ?))', ['test', False, 1729469991.993238])
frigate  | 2024-10-21 00:28:12.602116324  [2024-10-21 00:28:12] frigate.util.object            DEBUG   : Found 0 new events for test
frigate  | 2024-10-21 00:28:12.608285831  [2024-10-21 00:28:12] frigate.app                    INFO    : Camera processor started for test: 429
frigate  | 2024-10-21 00:28:12.609042369  [2024-10-21 00:28:12] frigate.comms.mqtt             DEBUG   : MQTT connected
frigate  | 2024-10-21 00:28:12.612734549  [2024-10-21 00:28:12] tzlocal                        DEBUG   : /etc/timezone found, contents:
frigate  | 2024-10-21 00:28:12.612736092   Etc/UTC
frigate  | 2024-10-21 00:28:12.612736864
frigate  | 2024-10-21 00:28:12.613199798  [2024-10-21 00:28:12] tzlocal                        DEBUG   : /etc/localtime found
frigate  | 2024-10-21 00:28:12.614266962  [2024-10-21 00:28:12] tzlocal                        DEBUG   : 2 found:
frigate  | 2024-10-21 00:28:12.614268716   {'/etc/timezone': 'Etc/UTC', '/etc/localtime is a symlink to': 'Etc/UTC'}
frigate  | 2024-10-21 00:28:12.615344366  [2024-10-21 00:28:12] frigate.app                    INFO    : Capture process started for test: 435
frigate  | 2024-10-21 00:28:12.617941409  [2024-10-21 00:28:12] peewee                         DEBUG   : ('SELECT COUNT(?) FROM "recordings" AS "t1" WHERE (("t1"."camera" = ?) AND ("t1"."segment_size" > ?))', ['*', 'test', 0.0])
frigate  | 2024-10-21 00:28:12.620300723  [2024-10-21 00:28:12] urllib3.connectionpool         DEBUG   : Starting new HTTPS connection (1): api.github.com:443
frigate  | 2024-10-21 00:28:12.621011554  [2024-10-21 00:28:12] peewee                         DEBUG   : ('SELECT AVG("t1"."segment_size" / ("t1"."end_time" - "t1"."start_time")) FROM "recordings" AS "t1" WHERE (("t1"."camera" = ?) AND ("t1"."segment_size" > ?)) LIMIT ?', ['test', 0.0, 100])
frigate  | 2024-10-21 00:28:12.621257549  [2024-10-21 00:28:12] frigate.storage                DEBUG   : test has a bandwidth of 2277.34 MiB/hr.
frigate  | 2024-10-21 00:28:12.695133180  [2024-10-21 00:28:12] urllib3.connectionpool         DEBUG   : https://api.github.com:443 "GET /repos/blakeblackshear/frigate/releases/latest HTTP/11" 200 1573
frigate  | 2024-10-21 00:28:12.697388146  [2024-10-21 00:28:12] frigate.api.auth               DEBUG   : Using jwt secret from .jwt_secret file in config directory.
frigate  | 2024-10-21 00:28:12.723142144  [2024-10-21 00:28:12] peewee.sqliteq                 DEBUG   : received query UPDATE "reviewsegment" SET "end_time" = ? WHERE ("reviewsegment"."end_time" IS NULL)
frigate  | 2024-10-21 00:28:12.723197910  [2024-10-21 00:28:12] peewee                         DEBUG   : ('UPDATE "reviewsegment" SET "end_time" = ? WHERE ("reviewsegment"."end_time" IS NULL)', [1729470492.7198])
frigate  | 2024-10-21 00:28:12.723866852  [2024-10-21 00:28:12] peewee.sqliteq                 DEBUG   : received query UPDATE "event" SET "end_time" = ("event"."start_time" + ?) WHERE ("event"."end_time" IS NULL)
frigate  | 2024-10-21 00:28:12.723910585  [2024-10-21 00:28:12] peewee                         DEBUG   : ('UPDATE "event" SET "end_time" = ("event"."start_time" + ?) WHERE ("event"."end_time" IS NULL)', [30])
frigate  | 2024-10-21 00:28:12.724357639  [2024-10-21 00:28:12] peewee                         DEBUG   : ('SELECT COUNT(1) FROM (SELECT 1 FROM "user" AS "t1") AS "_wrapped"', [])
frigate  | 2024-10-21 00:28:13.918498721  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Num Available Devices: 1
frigate  | 2024-10-21 00:28:13.920412393  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::BatchedNMSDynamic_TRT version 1
frigate  | 2024-10-21 00:28:13.920446798  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::BatchedNMS_TRT version 1
frigate  | 2024-10-21 00:28:13.920489589  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::BatchTilePlugin_TRT version 1
frigate  | 2024-10-21 00:28:13.920523814  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::Clip_TRT version 1
frigate  | 2024-10-21 00:28:13.920552909  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::CoordConvAC version 1
frigate  | 2024-10-21 00:28:13.920581483  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::CropAndResizeDynamic version 1
frigate  | 2024-10-21 00:28:13.920613122  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::CropAndResize version 1
frigate  | 2024-10-21 00:28:13.920663277  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::DecodeBbox3DPlugin version 1
frigate  | 2024-10-21 00:28:13.920710867  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::DetectionLayer_TRT version 1
frigate  | 2024-10-21 00:28:13.920758176  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::EfficientNMS_Explicit_TF_TRT version 1
frigate  | 2024-10-21 00:28:13.920791559  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::EfficientNMS_Implicit_TF_TRT version 1
frigate  | 2024-10-21 00:28:13.920828018  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::EfficientNMS_ONNX_TRT version 1
frigate  | 2024-10-21 00:28:13.920859447  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::EfficientNMS_TRT version 1
frigate  | 2024-10-21 00:28:13.920905524  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::FlattenConcat_TRT version 1
frigate  | 2024-10-21 00:28:13.920944979  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::fMHA_V2 version 1
frigate  | 2024-10-21 00:28:13.920985946  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::fMHCA version 1
frigate  | 2024-10-21 00:28:13.921024018  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::GenerateDetection_TRT version 1
frigate  | 2024-10-21 00:28:13.921054215  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::GridAnchor_TRT version 1
frigate  | 2024-10-21 00:28:13.921091215  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::GridAnchorRect_TRT version 1
frigate  | 2024-10-21 00:28:13.921124588  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::GroupNorm version 1
frigate  | 2024-10-21 00:28:13.921156298  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::InstanceNormalization_TRT version 1
frigate  | 2024-10-21 00:28:13.921194239  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::InstanceNormalization_TRT version 2
frigate  | 2024-10-21 00:28:13.921224497  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::LayerNorm version 1
frigate  | 2024-10-21 00:28:13.921259503  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::LReLU_TRT version 1
frigate  | 2024-10-21 00:28:13.921295761  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::MultilevelCropAndResize_TRT version 1
frigate  | 2024-10-21 00:28:13.921346176  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::MultilevelProposeROI_TRT version 1
frigate  | 2024-10-21 00:28:13.921376544  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::MultiscaleDeformableAttnPlugin_TRT version 1
frigate  | 2024-10-21 00:28:13.921420807  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::NMSDynamic_TRT version 1
frigate  | 2024-10-21 00:28:13.921451375  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::NMS_TRT version 1
frigate  | 2024-10-21 00:28:13.921486060  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::Normalize_TRT version 1
frigate  | 2024-10-21 00:28:13.921514594  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::PillarScatterPlugin version 1
frigate  | 2024-10-21 00:28:13.921544901  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::PriorBox_TRT version 1
frigate  | 2024-10-21 00:28:13.921583134  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::ProposalDynamic version 1
frigate  | 2024-10-21 00:28:13.921615845  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::ProposalLayer_TRT version 1
frigate  | 2024-10-21 00:28:13.921649108  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::Proposal version 1
frigate  | 2024-10-21 00:28:13.921686719  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::PyramidROIAlign_TRT version 1
frigate  | 2024-10-21 00:28:13.921715894  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::Region_TRT version 1
frigate  | 2024-10-21 00:28:13.921749327  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::Reorg_TRT version 1
frigate  | 2024-10-21 00:28:13.921784153  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::ResizeNearest_TRT version 1
frigate  | 2024-10-21 00:28:13.921814100  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::ROIAlign_TRT version 1
frigate  | 2024-10-21 00:28:13.921839648  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::RPROI_TRT version 1
frigate  | 2024-10-21 00:28:13.921882318  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::ScatterND version 1
frigate  | 2024-10-21 00:28:13.921912425  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::SeqLen2Spatial version 1
frigate  | 2024-10-21 00:28:13.921947632  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::SpecialSlice_TRT version 1
frigate  | 2024-10-21 00:28:13.921975364  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::SplitGeLU version 1
frigate  | 2024-10-21 00:28:13.922002205  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::Split version 1
frigate  | 2024-10-21 00:28:13.922040778  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::VoxelGeneratorPlugin version 1
frigate  | 2024-10-21 00:28:13.962302272  [2024-10-21 00:28:13] frigate.detectors.plugins.tensorrt INFO    : Loaded engine size: 72 MiB
frigate  | 2024-10-21 00:28:14.043004490  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt DEBUG   : Trying to load shared library libcublas.so.11
frigate  | 2024-10-21 00:28:14.043045287  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt DEBUG   : Loaded shared library libcublas.so.11
frigate  | 2024-10-21 00:28:14.054779098  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt DEBUG   : Using cublas as plugin tactic source
frigate  | 2024-10-21 00:28:14.054816128  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt DEBUG   : Trying to load shared library libcublasLt.so.11
frigate  | 2024-10-21 00:28:14.054860051  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt DEBUG   : Loaded shared library libcublasLt.so.11
frigate  | 2024-10-21 00:28:14.054893774  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt DEBUG   : Using cublasLt as core library tactic source
frigate  | 2024-10-21 00:28:14.054935583  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +6, GPU +8, now: CPU 209, GPU 731 (MiB)
frigate  | 2024-10-21 00:28:14.054968936  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt DEBUG   : Trying to load shared library libcudnn.so.8
frigate  | 2024-10-21 00:28:14.055020203  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt DEBUG   : Loaded shared library libcudnn.so.8
frigate  | 2024-10-21 00:28:14.055064977  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt DEBUG   : Using cuDNN as plugin tactic source
frigate  | 2024-10-21 00:28:14.060761752  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt DEBUG   : Using cuDNN as core library tactic source
frigate  | 2024-10-21 00:28:14.060831483  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +2, GPU +10, now: CPU 211, GPU 741 (MiB)
frigate  | 2024-10-21 00:28:14.062876875  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt DEBUG   : Deserialization required 41379 microseconds.
frigate  | 2024-10-21 00:28:14.074169112  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +71, now: CPU 0, GPU 71 (MiB)
frigate  | 2024-10-21 00:28:14.074197535  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt DEBUG   : Trying to load shared library libcublas.so.11
frigate  | 2024-10-21 00:28:14.074233664  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt DEBUG   : Loaded shared library libcublas.so.11
frigate  | 2024-10-21 00:28:14.074264873  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt DEBUG   : Using cublas as plugin tactic source
frigate  | 2024-10-21 00:28:14.074301722  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt DEBUG   : Trying to load shared library libcublasLt.so.11
frigate  | 2024-10-21 00:28:14.074362497  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt DEBUG   : Loaded shared library libcublasLt.so.11
frigate  | 2024-10-21 00:28:14.074416449  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt DEBUG   : Using cublasLt as core library tactic source
frigate  | 2024-10-21 00:28:14.074462135  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +1, GPU +8, now: CPU 140, GPU 733 (MiB)
frigate  | 2024-10-21 00:28:14.076063368  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt DEBUG   : Trying to load shared library libcudnn.so.8
frigate  | 2024-10-21 00:28:14.076096781  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt DEBUG   : Loaded shared library libcudnn.so.8
frigate  | 2024-10-21 00:28:14.076128781  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt DEBUG   : Using cuDNN as plugin tactic source
frigate  | 2024-10-21 00:28:14.076173115  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt DEBUG   : Using cuDNN as core library tactic source
frigate  | 2024-10-21 00:28:14.076221907  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 140, GPU 741 (MiB)
frigate  | 2024-10-21 00:28:14.076273885  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt DEBUG   : Total per-runner device persistent memory is 662016
frigate  | 2024-10-21 00:28:14.076325993  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt DEBUG   : Total per-runner host persistent memory is 256304
frigate  | 2024-10-21 00:28:14.076361891  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt DEBUG   : Allocated activation device memory of size 14950400
frigate  | 2024-10-21 00:28:14.076415623  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +14, now: CPU 0, GPU 85 (MiB)
frigate  | 2024-10-21 00:28:14.076454646  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt DEBUG   : CUDA lazy loading is enabled.
frigate  | 2024-10-21 00:28:14.076500353  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt DEBUG   : Allocated Tensor Binding input Memory 1228800 Bytes (307200 * DataType.FLOAT)
frigate  | 2024-10-21 00:28:14.076530399  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt DEBUG   : Input has Shape (1, 3, 320, 320)
frigate  | 2024-10-21 00:28:14.076574573  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt DEBUG   : Allocated Tensor Binding detections Memory 176400 Bytes (44100 * DataType.FLOAT)
frigate  | 2024-10-21 00:28:14.076616001  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt DEBUG   : Output has Shape (1, 44100, 1, 1)
frigate  | 2024-10-21 00:28:14.076662719  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt DEBUG   : TensorRT loaded. Input shape is ((320, 320), <class 'numpy.float32'>)
frigate  | 2024-10-21 00:28:14.076696643  [2024-10-21 00:28:14] frigate.detectors.plugins.tensorrt DEBUG   : TensorRT version is 8
frigate  | 2024-10-21 00:28:17.580279162  [2024-10-21 00:28:17] asyncio                        DEBUG   : Using selector: EpollSelector
frigate  | 2024-10-21 00:28:21.476734073  [INFO] Starting go2rtc healthcheck service...
frigate  | 2024-10-21 00:28:22.586065690  [2024-10-21 00:28:22] asyncio                        DEBUG   : Using selector: EpollSelector
frigate  | 2024-10-21 00:28:27.591492727  [2024-10-21 00:28:27] asyncio                        DEBUG   : Using selector: EpollSelector
frigate  | 2024-10-21 00:28:27.773666990  [2024-10-21 00:28:27] frigate.record.maintainer      DEBUG   : Copied /media/frigate/recordings/2024-10-21/00/test/28.13.mp4 in 0.09169912338256836 seconds.
frigate  | 2024-10-21 00:28:27.775477949  [2024-10-21 00:28:27] peewee.sqliteq                 DEBUG   : received query INSERT INTO "recordings" ("id", "camera", "path", "start_time", "end_time", "duration", "motion", "objects", "dBFS", "segment_size", "regions") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
frigate  | 2024-10-21 00:28:27.775538102  [2024-10-21 00:28:27] peewee                         DEBUG   : ('INSERT INTO "recordings" ("id", "camera", "path", "start_time", "end_time", "duration", "motion", "objects", "dBFS", "segment_size", "regions") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)', ['1729470493.0-ojpadz', 'test', '/media/frigate/recordings/2024-10-21/00/test/28.13.mp4', 1729470493.0, 1729470502.947956, 9.947955555555556, 312, 0, 0, 6.3, 142])
frigate  | 2024-10-21 00:28:32.596699425  [2024-10-21 00:28:32] asyncio                        DEBUG   : Using selector: EpollSelector
frigate  | 2024-10-21 00:28:37.311123612  192.168.40.100 - - [21/Oct/2024:00:28:37 +0000] "GET /api/config HTTP/1.1" 200 7787 "http://192.168.40.15:5000/settings" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:131.0) Gecko/20100101 Firefox/131.0" "-"
frigate  | 2024-10-21 00:28:37.311128301  192.168.40.100 - - [21/Oct/2024:00:28:37 +0000] "GET /api/profile HTTP/1.1" 200 18 "http://192.168.40.15:5000/settings" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:131.0) Gecko/20100101 Firefox/131.0" "-"
frigate  | 2024-10-21 00:28:37.600824320  [2024-10-21 00:28:37] asyncio                        DEBUG   : Using selector: EpollSelector
frigate  | 2024-10-21 00:28:37.706336823  [2024-10-21 00:28:37] frigate.stats.emitter          DEBUG   : Starting stats collection
frigate  | 2024-10-21 00:28:37.706384383  [2024-10-21 00:28:37] asyncio                        DEBUG   : Using selector: EpollSelector
frigate  | 2024-10-21 00:28:37.736140866  [2024-10-21 00:28:37] frigate.stats.emitter          DEBUG   : Finished stats collection
frigate  | 2024-10-21 00:28:37.773493810  [2024-10-21 00:28:37] frigate.record.maintainer      DEBUG   : Copied /media/frigate/recordings/2024-10-21/00/test/28.23.mp4 in 0.08484983444213867 seconds.
frigate  | 2024-10-21 00:28:37.775338833  [2024-10-21 00:28:37] peewee.sqliteq                 DEBUG   : received query INSERT INTO "recordings" ("id", "camera", "path", "start_time", "end_time", "duration", "motion", "objects", "dBFS", "segment_size", "regions") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
frigate  | 2024-10-21 00:28:37.775407503  [2024-10-21 00:28:37] peewee                         DEBUG   : ('INSERT INTO "recordings" ("id", "camera", "path", "start_time", "end_time", "duration", "motion", "objects", "dBFS", "segment_size", "regions") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)', ['1729470503.0-1zdjui', 'test', '/media/frigate/recordings/2024-10-21/00/test/28.23.mp4', 1729470503.0, 1729470512.925544, 9.925544444444444, 847, 0, 0, 6.3, 524])
frigate  | 2024-10-21 00:28:38.905122936  192.168.40.100 - - [21/Oct/2024:00:28:38 +0000] "GET /settings HTTP/1.1" 200 2651 "http://192.168.40.15:5000/settings" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:131.0) Gecko/20100101 Firefox/131.0" "-"
frigate  | 2024-10-21 00:28:39.001002793  192.168.40.100 - - [21/Oct/2024:00:28:39 +0000] "GET /api/profile HTTP/1.1" 200 18 "http://192.168.40.15:5000/settings" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:131.0) Gecko/20100101 Firefox/131.0" "-"
frigate  | 2024-10-21 00:28:39.001006029  192.168.40.100 - - [21/Oct/2024:00:28:39 +0000] "GET /api/config HTTP/1.1" 200 7787 "http://192.168.40.15:5000/settings" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:131.0) Gecko/20100101 Firefox/131.0" "-"
frigate  | 2024-10-21 00:28:39.018631843  [2024-10-21 00:28:39] frigate.comms.ws               DEBUG   : Publishing mqtt message from websockets at onConnect.
frigate  | 2024-10-21 00:28:39.027046758  192.168.40.100 - - [21/Oct/2024:00:28:39 +0000] "GET /images/apple-touch-icon.png HTTP/1.1" 200 4042 "http://192.168.40.15:5000/settings" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:131.0) Gecko/20100101 Firefox/131.0" "-"
frigate  | 2024-10-21 00:28:42.606516781  [2024-10-21 00:28:42] asyncio                        DEBUG   : Using selector: EpollSelector
frigate  | 2024-10-21 00:28:47.611344309  [2024-10-21 00:28:47] asyncio                        DEBUG   : Using selector: EpollSelector
frigate  | 2024-10-21 00:28:47.779322553  [2024-10-21 00:28:47] frigate.record.maintainer      DEBUG   : Copied /media/frigate/recordings/2024-10-21/00/test/28.33.mp4 in 0.0814809799194336 seconds.
frigate  | 2024-10-21 00:28:47.781212480  [2024-10-21 00:28:47] peewee.sqliteq                 DEBUG   : received query INSERT INTO "recordings" ("id", "camera", "path", "start_time", "end_time", "duration", "motion", "objects", "dBFS", "segment_size", "regions") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
frigate  | 2024-10-21 00:28:47.781257115  [2024-10-21 00:28:47] peewee                         DEBUG   : ('INSERT INTO "recordings" ("id", "camera", "path", "start_time", "end_time", "duration", "motion", "objects", "dBFS", "segment_size", "regions") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)', ['1729470513.0-57cx6l', 'test', '/media/frigate/recordings/2024-10-21/00/test/28.33.mp4', 1729470513.0, 1729470522.934578, 9.934577777777779, 208, 0, 0, 6.3, 154])
frigate  | 2024-10-21 00:28:48.252584680  192.168.40.100 - - [21/Oct/2024:00:28:48 +0000] "GET /fonts/Inter-Italic.woff2 HTTP/1.1" 200 114576 "http://192.168.40.15:5000/assets/index-BsqM9WWT.css" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:131.0) Gecko/20100101 Firefox/131.0" "-"
frigate  | 2024-10-21 00:28:48.255053441  192.168.40.100 - - [21/Oct/2024:00:28:48 +0000] "GET /api/config HTTP/1.1" 200 7787 "http://192.168.40.15:5000/settings" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:131.0) Gecko/20100101 Firefox/131.0" "-"
frigate  | 2024-10-21 00:28:49.857747439  192.168.40.100 - - [21/Oct/2024:00:28:49 +0000] "GET /api/test/latest.webp?cache=1729470529777 HTTP/1.1" 200 77166 "http://192.168.40.15:5000/settings" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:131.0) Gecko/20100101 Firefox/131.0" "-"
frigate  | 2024-10-21 00:28:52.132608141  [2024-10-21 00:28:52] frigate.comms.ws               DEBUG   : Publishing mqtt message from websockets at test/motion_threshold/set.
frigate  | 2024-10-21 00:28:52.132662924  [2024-10-21 00:28:52] frigate.comms.dispatcher       INFO    : Setting motion threshold for test: 30




crash with yolov4-608:
frigate  | 2024-10-21 00:19:56.973024780  [2024-10-21 00:19:56] asyncio                        DEBUG   : Using selector: EpollSelector
frigate  | 2024-10-21 00:20:01.978544939  [2024-10-21 00:20:01] asyncio                        DEBUG   : Using selector: EpollSelector
frigate  | 2024-10-21 00:20:06.983596122  [2024-10-21 00:20:06] asyncio                        DEBUG   : Using selector: EpollSelector
frigate  | 2024-10-21 00:20:11.988679518  [2024-10-21 00:20:11] frigate.record.maintainer      DEBUG   : Found 3 old frames.
frigate  | 2024-10-21 00:20:11.988723181  [2024-10-21 00:20:11] asyncio                        DEBUG   : Using selector: EpollSelector
frigate  | 2024-10-21 00:20:12.075829364  [2024-10-21 00:20:12] frigate.record.maintainer      DEBUG   : Copied /media/frigate/recordings/2024-10-21/00/test/19.53.mp4 in 0.08397197723388672 seconds.
frigate  | 2024-10-21 00:20:12.077521709  [2024-10-21 00:20:12] peewee.sqliteq                 DEBUG   : received query INSERT INTO "recordings" ("id", "camera", "path", "start_time", "end_time", "duration", "motion", "objects", "dBFS", "segment_size", "regions") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
frigate  | 2024-10-21 00:20:12.077572515  [2024-10-21 00:20:12] peewee                         DEBUG   : ('INSERT INTO "recordings" ("id", "camera", "path", "start_time", "end_time", "duration", "motion", "objects", "dBFS", "segment_size", "regions") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)', ['1729469993.0-s9tay4', 'test', '/media/frigate/recordings/2024-10-21/00/test/19.53.mp4', 1729469993.0, 1729470002.945133, 9.945133333333333, 2, 0, 0, 6.3, 0])
frigate  | 2024-10-21 00:20:12.138172551  [2024-10-21 00:20:12] frigate.watchdog               INFO    : Detection appears to be stuck. Restarting detection process...
frigate  | 2024-10-21 00:20:12.158464876  [2024-10-21 00:20:12] detector.tensorrt              INFO    : Starting detection process: 1006
frigate  | 2024-10-21 00:20:12.233187427  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Num Available Devices: 1
frigate  | 2024-10-21 00:20:12.239750268  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::BatchedNMSDynamic_TRT version 1
frigate  | 2024-10-21 00:20:12.239786066  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::BatchedNMS_TRT version 1
frigate  | 2024-10-21 00:20:12.239824198  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::BatchTilePlugin_TRT version 1
frigate  | 2024-10-21 00:20:12.239857721  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::Clip_TRT version 1
frigate  | 2024-10-21 00:20:12.239887668  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::CoordConvAC version 1
frigate  | 2024-10-21 00:20:12.239917033  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::CropAndResizeDynamic version 1
frigate  | 2024-10-21 00:20:12.239945277  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::CropAndResize version 1
frigate  | 2024-10-21 00:20:12.239974241  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::DecodeBbox3DPlugin version 1
frigate  | 2024-10-21 00:20:12.240002404  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::DetectionLayer_TRT version 1
frigate  | 2024-10-21 00:20:12.240030708  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::EfficientNMS_Explicit_TF_TRT version 1
frigate  | 2024-10-21 00:20:12.240087635  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::EfficientNMS_Implicit_TF_TRT version 1
frigate  | 2024-10-21 00:20:12.240119205  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::EfficientNMS_ONNX_TRT version 1
frigate  | 2024-10-21 00:20:12.240151867  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::EfficientNMS_TRT version 1
frigate  | 2024-10-21 00:20:12.240181453  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::FlattenConcat_TRT version 1
frigate  | 2024-10-21 00:20:12.240209606  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::fMHA_V2 version 1
frigate  | 2024-10-21 00:20:12.240238921  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::fMHCA version 1
frigate  | 2024-10-21 00:20:12.240265892  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::GenerateDetection_TRT version 1
frigate  | 2024-10-21 00:20:12.240294266  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::GridAnchor_TRT version 1
frigate  | 2024-10-21 00:20:12.240336024  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::GridAnchorRect_TRT version 1
frigate  | 2024-10-21 00:20:12.240369047  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::GroupNorm version 1
frigate  | 2024-10-21 00:20:12.240420113  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::InstanceNormalization_TRT version 1
frigate  | 2024-10-21 00:20:12.240468555  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::InstanceNormalization_TRT version 2
frigate  | 2024-10-21 00:20:12.240522226  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::LayerNorm version 1
frigate  | 2024-10-21 00:20:12.240568864  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::LReLU_TRT version 1
frigate  | 2024-10-21 00:20:12.240622064  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::MultilevelCropAndResize_TRT version 1
frigate  | 2024-10-21 00:20:12.240670827  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::MultilevelProposeROI_TRT version 1
frigate  | 2024-10-21 00:20:12.240711403  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::MultiscaleDeformableAttnPlugin_TRT version 1
frigate  | 2024-10-21 00:20:12.240754074  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::NMSDynamic_TRT version 1
frigate  | 2024-10-21 00:20:12.240799730  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::NMS_TRT version 1
frigate  | 2024-10-21 00:20:12.240832552  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::Normalize_TRT version 1
frigate  | 2024-10-21 00:20:12.240875954  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::PillarScatterPlugin version 1
frigate  | 2024-10-21 00:20:12.240917913  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::PriorBox_TRT version 1
frigate  | 2024-10-21 00:20:12.240964641  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::ProposalDynamic version 1
frigate  | 2024-10-21 00:20:12.241007282  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::ProposalLayer_TRT version 1
frigate  | 2024-10-21 00:20:12.241049592  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::Proposal version 1
frigate  | 2024-10-21 00:20:12.241089627  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::PyramidROIAlign_TRT version 1
frigate  | 2024-10-21 00:20:12.241135484  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::Region_TRT version 1
frigate  | 2024-10-21 00:20:12.241175489  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::Reorg_TRT version 1
frigate  | 2024-10-21 00:20:12.241220504  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::ResizeNearest_TRT version 1
frigate  | 2024-10-21 00:20:12.241259909  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::ROIAlign_TRT version 1
frigate  | 2024-10-21 00:20:12.241294634  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::RPROI_TRT version 1
frigate  | 2024-10-21 00:20:12.241352594  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::ScatterND version 1
frigate  | 2024-10-21 00:20:12.241393661  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::SeqLen2Spatial version 1
frigate  | 2024-10-21 00:20:12.241442423  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::SpecialSlice_TRT version 1
frigate  | 2024-10-21 00:20:12.241480335  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::SplitGeLU version 1
frigate  | 2024-10-21 00:20:12.241517976  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::Split version 1
frigate  | 2024-10-21 00:20:12.241565376  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::VoxelGeneratorPlugin version 1
frigate  | 2024-10-21 00:20:12.296248471  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt INFO    : Loaded engine size: 125 MiB
frigate  | 2024-10-21 00:20:12.398767244  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Trying to load shared library libcublas.so.11
frigate  | 2024-10-21 00:20:12.398809414  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Loaded shared library libcublas.so.11
frigate  | 2024-10-21 00:20:12.412177394  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Using cublas as plugin tactic source
frigate  | 2024-10-21 00:20:12.412213863  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Trying to load shared library libcublasLt.so.11
frigate  | 2024-10-21 00:20:12.412258357  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Loaded shared library libcublasLt.so.11
frigate  | 2024-10-21 00:20:12.412335362  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Using cublasLt as core library tactic source
frigate  | 2024-10-21 00:20:12.412387130  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +6, GPU +8, now: CPU 271, GPU 787 (MiB)
frigate  | 2024-10-21 00:20:12.412437305  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Trying to load shared library libcudnn.so.8
frigate  | 2024-10-21 00:20:12.412504782  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Loaded shared library libcudnn.so.8
frigate  | 2024-10-21 00:20:12.412577820  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Using cuDNN as plugin tactic source
frigate  | 2024-10-21 00:20:12.418377310  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Using cuDNN as core library tactic source
frigate  | 2024-10-21 00:20:12.418513186  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +1, GPU +10, now: CPU 272, GPU 797 (MiB)
frigate  | 2024-10-21 00:20:12.420966899  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Deserialization required 50736 microseconds.
frigate  | 2024-10-21 00:20:12.427032862  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +126, now: CPU 0, GPU 126 (MiB)
frigate  | 2024-10-21 00:20:12.437657713  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Trying to load shared library libcublas.so.11
frigate  | 2024-10-21 00:20:12.437698660  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Loaded shared library libcublas.so.11
frigate  | 2024-10-21 00:20:12.437750678  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Using cublas as plugin tactic source
frigate  | 2024-10-21 00:20:12.437798709  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Trying to load shared library libcublasLt.so.11
frigate  | 2024-10-21 00:20:12.437856498  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Loaded shared library libcublasLt.so.11
frigate  | 2024-10-21 00:20:12.437904308  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Using cublasLt as core library tactic source
frigate  | 2024-10-21 00:20:12.437952439  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +1, GPU +8, now: CPU 148, GPU 791 (MiB)
frigate  | 2024-10-21 00:20:12.437995420  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Trying to load shared library libcudnn.so.8
frigate  | 2024-10-21 00:20:12.438654675  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Loaded shared library libcudnn.so.8
frigate  | 2024-10-21 00:20:12.438697937  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Using cuDNN as plugin tactic source
frigate  | 2024-10-21 00:20:12.438746939  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Using cuDNN as core library tactic source
frigate  | 2024-10-21 00:20:12.438788127  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 148, GPU 799 (MiB)
frigate  | 2024-10-21 00:20:12.438842620  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Total per-runner device persistent memory is 3787264
frigate  | 2024-10-21 00:20:12.438887144  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Total per-runner host persistent memory is 390032
frigate  | 2024-10-21 00:20:12.438925366  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Allocated activation device memory of size 66169856
frigate  | 2024-10-21 00:20:12.438966894  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +67, now: CPU 0, GPU 193 (MiB)
frigate  | 2024-10-21 00:20:12.439006990  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : CUDA lazy loading is enabled.
frigate  | 2024-10-21 00:20:12.439054370  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Allocated Tensor Binding input Memory 4435968 Bytes (1108992 * DataType.FLOAT)
frigate  | 2024-10-21 00:20:12.439101298  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Input has Shape (1, 3, 608, 608)
frigate  | 2024-10-21 00:20:12.439144580  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Allocated Tensor Binding detections Memory 636804 Bytes (159201 * DataType.FLOAT)
frigate  | 2024-10-21 00:20:12.439184956  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : Output has Shape (1, 159201, 1, 1)
frigate  | 2024-10-21 00:20:12.439230843  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : TensorRT loaded. Input shape is ((608, 608), <class 'numpy.float32'>)
frigate  | 2024-10-21 00:20:12.439278853  [2024-10-21 00:20:12] frigate.detectors.plugins.tensorrt DEBUG   : TensorRT version is 8
frigate  | 2024-10-21 00:20:12.440174615  Fatal Python error: Segmentation fault
frigate  | 2024-10-21 00:20:12.440175196
frigate  | 2024-10-21 00:20:12.440184644  Thread 0x000075590a0006c0 (most recent call first):
frigate  | 2024-10-21 00:20:12.440196095    File "/usr/lib/python3.9/threading.py", line 312 in wait
frigate  | 2024-10-21 00:20:12.440222235    File "/usr/lib/python3.9/multiprocessing/queues.py", line 233 in _feed
frigate  | 2024-10-21 00:20:12.440241882    File "/usr/lib/python3.9/threading.py", line 892 in run
frigate  | 2024-10-21 00:20:12.440267781    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
frigate  | 2024-10-21 00:20:12.440291435    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
frigate  | 2024-10-21 00:20:12.440291876
frigate  | 2024-10-21 00:20:12.440299981  Current thread 0x00007559096006c0 (most recent call first):
frigate  | 2024-10-21 00:20:12.440324658    File "/opt/frigate/frigate/detectors/plugins/tensorrt.py", line 196 in <listcomp>
frigate  | 2024-10-21 00:20:12.440358181    File "/opt/frigate/frigate/detectors/plugins/tensorrt.py", line 195 in _do_inference
frigate  | 2024-10-21 00:20:12.440386304    File "/opt/frigate/frigate/detectors/plugins/tensorrt.py", line 312 in detect_raw
frigate  | 2024-10-21 00:20:12.440409208    File "/opt/frigate/frigate/object_detection.py", line 75 in detect_raw
frigate  | 2024-10-21 00:20:12.440433183    File "/opt/frigate/frigate/object_detection.py", line 125 in run_detector
frigate  | 2024-10-21 00:20:12.440454633    File "/usr/lib/python3.9/multiprocessing/process.py", line 108 in run
frigate  | 2024-10-21 00:20:12.440482015    File "/usr/lib/python3.9/multiprocessing/process.py", line 315 in _bootstrap
frigate  | 2024-10-21 00:20:12.440505099    File "/usr/lib/python3.9/multiprocessing/popen_fork.py", line 71 in _launch
frigate  | 2024-10-21 00:20:12.440532631    File "/usr/lib/python3.9/multiprocessing/popen_fork.py", line 19 in __init__
frigate  | 2024-10-21 00:20:12.440555674    File "/usr/lib/python3.9/multiprocessing/context.py", line 277 in _Popen
frigate  | 2024-10-21 00:20:12.440580531    File "/usr/lib/python3.9/multiprocessing/context.py", line 224 in _Popen
frigate  | 2024-10-21 00:20:12.440603665    File "/usr/lib/python3.9/multiprocessing/process.py", line 121 in start
frigate  | 2024-10-21 00:20:12.440630966    File "/opt/frigate/frigate/object_detection.py", line 183 in start_or_restart
frigate  | 2024-10-21 00:20:12.440655342    File "/opt/frigate/frigate/watchdog.py", line 34 in run
frigate  | 2024-10-21 00:20:12.440679518    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
frigate  | 2024-10-21 00:20:12.440702071    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap

Install method

Docker Compose

Object Detector

TensorRT

Screenshots of the Frigate UI's System metrics pages

Any other information that may be helpful

No response

NickM-27 · 2024-10-21T02:45:55Z

NickM-27
Oct 21, 2024
Collaborator Sponsor

You have important fields like the tensor input and pixel format commented our when it should not be

10 replies

nordic-style Oct 21, 2024
Author

I have just deleted everything and recreated all the models etc. Could it be that there is another problem? When I change to a larger model than 320, I also get a segmentation fault. The gpu is working fine (just tested again). This was with bgr and nchw:

NickM-27 Oct 21, 2024
Collaborator Sponsor

I would recommend trying with a bit more of a normal scenario as that might be a bit difficult to recognize as person

detectors:
  tensorrt:
    type: tensorrt
    device: 0 #This is the default, select the first GPU

model:
  path: /config/model_cache/tensorrt/yolov7-320.trt
  input_tensor: nchw
  input_pixel_format: rgb
  width: 320
  height: 320

is the config that should be used, there is no use in trying to change any of the parameters

nordic-style Oct 22, 2024
Author

I will use that settings:
'''
ffmpeg:
hwaccel_args: preset-nvidia-h264

detectors:
tensorrt:
type: tensorrt
device: 0
model:
path: /config/model_cache/tensorrt/yolov7-320.trt
input_tensor: nchw
input_pixel_format: rgb
width: 320
height: 320
'''

Regarding the scenario: i am running through the image and still have issues with the detection (none). I will integrate other cams for better samples. I still think something is wrong with my setup.

nordic-style Oct 22, 2024
Author

I will try the next version of the frigate. Hope that it will work better so that I can use frigate+ model training which sounds great ...

NickM-27 Oct 22, 2024
Collaborator Sponsor

yeah, you would use onnx models which work entirely differently than what you are doing now

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Detector Support]: 4070 ti detection does not work (setup does) with yolov7-320 and crashes with bigger models. #14465

{{title}}

Replies: 1 comment 10 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

[Detector Support]: 4070 ti detection does not work (setup does) with yolov7-320 and crashes with bigger models. #14465

nordic-style Oct 21, 2024

Describe the problem you are having

Version

Frigate config file

docker-compose file or Docker CLI command

Relevant Frigate log output

Install method

Object Detector

Screenshots of the Frigate UI's System metrics pages

Any other information that may be helpful

Replies: 1 comment · 10 replies

NickM-27 Oct 21, 2024 Collaborator Sponsor

nordic-style Oct 21, 2024 Author

NickM-27 Oct 21, 2024 Collaborator Sponsor

nordic-style Oct 22, 2024 Author

nordic-style Oct 22, 2024 Author

NickM-27 Oct 22, 2024 Collaborator Sponsor

nordic-style
Oct 21, 2024

Replies: 1 comment 10 replies

NickM-27
Oct 21, 2024
Collaborator Sponsor

nordic-style Oct 21, 2024
Author

NickM-27 Oct 21, 2024
Collaborator Sponsor

nordic-style Oct 22, 2024
Author

nordic-style Oct 22, 2024
Author

NickM-27 Oct 22, 2024
Collaborator Sponsor