[Detector Support]: TensorRT Crashing into Boot Loop #11144

JustJarethB · 2024-04-29T00:58:56Z

JustJarethB
Apr 29, 2024

Describe the problem you are having

Preface

Hi folks. I've been having a nightmare with getting my GPU drivers working nicely with [the whole stack] but TLDR: my GTX 1060 is now pushing hardware accel for decoding perfectly. I really want the GPU in to do detection though as I don't want to bake the CPU and I have no TPU

Issue

When adding to config.yml:

detectors: 
  tensorrt:
    type: tensorrt

frigate crashes and pushes docker into a boot loop (due to restart: unless-stopped)
Full logs below, but the rabbit hole I've been chasing is

AttributeError: 'TensorRtDetector' object has no attribute 'outputs'

which seems to suggest

.trt is not mapped [Config Support]: Can't get TensorRT to start up #5058
models need building [HW Accel Support]: "AttributeError: 'TensorRtDetector' object has no attribute 'outputs' I get this error #7659
I have tried to do both but clearly been unsucessful

Checklist

I have:

used frigate:stable-tensorrt image
installed CUDA and CRT and NVIDIA drivers
verified hardware accel with nvidia-smi
a GTX 1060
lost the will to continue

Version

0.13.2-6476f8a

Frigate config file

logger:
  default: debug
mqtt:
  # Optional: Enable mqtt server (default: shown below)
  enabled: false

# Include all cameras by default in Birdseye view
birdseye:
  enabled: True
  mode: continuous

# detectors: 
#   tensorrt:
#     type: tensorrt
    # device: 0 #This is the default, select the first GPU

# model:
#   path: /config/model_cache/tensorrt/yolov7-tiny-320.trt
#   input_tensor: nchw
#   input_pixel_format: rgb
#   width: 320
#   height: 320

ffmpeg:
  # Optional: global ffmpeg args (default: shown below)
  global_args: -hide_banner -loglevel warning -threads 2
  # Optional: global hwaccel args (default: shown below)
  # NOTE: See hardware acceleration docs for your specific device
  hwaccel_args: preset-nvidia-h264

record:
  enabled: True
  retain:
    days: 28
    mode: all
  sync_recordings: True

cameras:
  office:
    detect:
      enabled: False
    record:
      enabled: True
    ffmpeg:
      inputs:
        - path: rtsp://FRIGATEu0:[email protected]:554/live0
          roles:
            - record
            - clips
  kitchen:
    detect:
      enabled: False
    record:
      enabled: True
    ffmpeg:
      inputs:
        - path: rtsp://FRIGATEu0:[email protected]:554/live0
          roles:
            - record
            - clips

docker-compose file or Docker CLI command

version: "3.9"
services:
  frigate:

    stdin_open: true # docker run -i
    tty: true        # docker run -t    container_name: frigate
    privileged: true # this may not be necessary for all setups
#    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:stable-tensorrt
    deploy:    # <------------- Add this section
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1 # number of GPUs
              capabilities: [gpu]    
    shm_size: "128mb" # update for your cameras based on calculation above
    devices:
      - /dev/bus/usb:/dev/bus/usb  # Passes the USB Coral, needs to be modified for other versions
      - /dev/apex_0:/dev/apex_0    # Passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-on-linux
      - /dev/video11:/dev/video11  # For Raspberry Pi 4B
      - /dev/dri/renderD128:/dev/dri/renderD128 # For intel hwaccel, needs to be updated for your hardware
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /home/jareth/docker/volumes/frigate/config:/config
      - /home/jareth/docker/volumes/frigate/storage:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "5000:5000"
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
    environment:
      FRIGATE_RTSP_PASSWORD: "password"
      USE_FP16: False

Relevant log output

s6-rc: info: service s6rc-fdholder: starting
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service s6rc-fdholder successfully started
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service trt-model-prepare: starting
s6-rc: info: service log-prepare: starting
s6-rc: info: service log-prepare successfully started
No models to convert.
s6-rc: info: service nginx-log: starting
s6-rc: info: service go2rtc-log: starting
s6-rc: info: service frigate-log: starting
s6-rc: info: service trt-model-prepare successfully started
s6-rc: info: service nginx-log successfully started
s6-rc: info: service go2rtc-log successfully started
s6-rc: info: service go2rtc: starting
s6-rc: info: service frigate-log successfully started
s6-rc: info: service go2rtc successfully started
s6-rc: info: service go2rtc-healthcheck: starting
s6-rc: info: service frigate: starting
s6-rc: info: service frigate successfully started
s6-rc: info: service nginx: starting
s6-rc: info: service go2rtc-healthcheck successfully started
2024-04-29 01:41:49.737008432  [INFO] Preparing Frigate...
s6-rc: info: service nginx successfully started
s6-rc: info: service legacy-services: starting
2024-04-29 01:41:49.740306193  [INFO] Preparing new go2rtc config...
2024-04-29 01:41:49.740413153  [INFO] Starting NGINX...
2024-04-29 01:41:49.749334042  [INFO] Starting Frigate...
s6-rc: info: service legacy-services successfully started
2024-04-29 01:41:49.961871598  [INFO] Starting go2rtc...
2024-04-29 01:41:50.090228834  01:41:50.090 INF go2rtc version 1.8.4 linux/amd64
2024-04-29 01:41:50.090581134  01:41:50.090 INF [rtsp] listen addr=:8554
2024-04-29 01:41:50.090729275  01:41:50.090 INF [webrtc] listen addr=:8555
2024-04-29 01:41:50.091166509  01:41:50.091 INF [api] listen addr=:1984
2024-04-29 01:41:51.030828571  [2024-04-29 01:41:51] frigate.app                    INFO    : Starting Frigate (0.13.2-6476f8a)
2024-04-29 01:41:52.598605005  [2024-04-29 01:41:52] frigate.app                    INFO    : Recording process started: 312
2024-04-29 01:41:52.600982787  [2024-04-29 01:41:52] frigate.app                    INFO    : go2rtc process pid: 98
2024-04-29 01:41:52.604574567  [2024-04-29 01:41:52] asyncio                        DEBUG   : Using selector: EpollSelector
2024-04-29 01:41:52.621725709  [2024-04-29 01:41:52] frigate.app                    INFO    : Output process started: 324
2024-04-29 01:41:52.627631844  [2024-04-29 01:41:52] frigate.util.object            DEBUG   : Found 0 new events for office
2024-04-29 01:41:52.629288049  [2024-04-29 01:41:52] frigate.util.object            DEBUG   : Found 0 new events for kitchen
2024-04-29 01:41:52.634133487  [2024-04-29 01:41:52] frigate.app                    INFO    : Camera processor started for office: 329
2024-04-29 01:41:52.639104760  [2024-04-29 01:41:52] frigate.app                    INFO    : Camera processor started for kitchen: 330
2024-04-29 01:41:52.645424689  [2024-04-29 01:41:52] tzlocal                        DEBUG   : /etc/timezone found, contents:
2024-04-29 01:41:52.645428874   Etc/UTC
2024-04-29 01:41:52.645430369  
2024-04-29 01:41:52.645925213  [2024-04-29 01:41:52] tzlocal                        DEBUG   : /etc/localtime found
2024-04-29 01:41:52.646136755  [2024-04-29 01:41:52] tzlocal                        DEBUG   : 2 found:
2024-04-29 01:41:52.646139551   {'/etc/timezone': 'Etc/UTC', '/etc/localtime is a symlink to': 'Etc/UTC'}
2024-04-29 01:41:52.652361475  [2024-04-29 01:41:52] frigate.app                    INFO    : Capture process started for office: 332
2024-04-29 01:41:52.652578405  [2024-04-29 01:41:52] frigate.app                    INFO    : Capture process started for kitchen: 334
2024-04-29 01:41:52.663358739  [2024-04-29 01:41:52] urllib3.connectionpool         DEBUG   : Starting new HTTPS connection (1): api.github.com:443
2024-04-29 01:41:52.664536410  [2024-04-29 01:41:52] frigate.storage                DEBUG   : office has a bandwidth of 303.11 MiB/hr.
2024-04-29 01:41:52.670111888  [2024-04-29 01:41:52] frigate.storage                DEBUG   : kitchen has a bandwidth of 236.22 MiB/hr.
2024-04-29 01:41:52.696500120  [2024-04-29 01:41:52] tzlocal                        DEBUG   : /etc/timezone found, contents:
2024-04-29 01:41:52.696504213   Etc/UTC
2024-04-29 01:41:52.696505740  
2024-04-29 01:41:52.697250127  [2024-04-29 01:41:52] tzlocal                        DEBUG   : /etc/localtime found
2024-04-29 01:41:52.698808945  [2024-04-29 01:41:52] tzlocal                        DEBUG   : 2 found:
2024-04-29 01:41:52.698812961   {'/etc/timezone': 'Etc/UTC', '/etc/localtime is a symlink to': 'Etc/UTC'}
2024-04-29 01:41:52.748754528  [2024-04-29 01:41:52] urllib3.connectionpool         DEBUG   : https://api.github.com:443 "GET /repos/blakeblackshear/frigate/releases/latest HTTP/1.1" 403 279
2024-04-29 01:41:52.775194656  [2024-04-29 01:41:52] frigate.record.util            DEBUG   : Start sync recordings.
2024-04-29 01:41:52.934919352  [2024-04-29 01:41:52] detector.tensorrt              INFO    : Starting detection process: 322
2024-04-29 01:41:53.024011347  [2024-04-29 01:41:52] frigate.detectors.plugins.tensorrt DEBUG   : Num Available Devices: 1
2024-04-29 01:41:53.024334096  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::BatchedNMSDynamic_TRT version 1
2024-04-29 01:41:53.032052750  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::BatchedNMS_TRT version 1
2024-04-29 01:41:53.032056873  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::BatchTilePlugin_TRT version 1
2024-04-29 01:41:53.032117909  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::Clip_TRT version 1
2024-04-29 01:41:53.032133748  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::CoordConvAC version 1
2024-04-29 01:41:53.032146443  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::CropAndResizeDynamic version 1
2024-04-29 01:41:53.032147711  Process detector:tensorrt:
2024-04-29 01:41:53.032151213  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::CropAndResize version 1
2024-04-29 01:41:53.032154784  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::DecodeBbox3DPlugin version 1
2024-04-29 01:41:53.032169041  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::DetectionLayer_TRT version 1
2024-04-29 01:41:53.032172581  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::EfficientNMS_Explicit_TF_TRT version 1
2024-04-29 01:41:53.032176098  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::EfficientNMS_Implicit_TF_TRT version 1
2024-04-29 01:41:53.032179558  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::EfficientNMS_ONNX_TRT version 1
2024-04-29 01:41:53.032180761  Traceback (most recent call last):
2024-04-29 01:41:53.032182135    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2024-04-29 01:41:53.032183207      self.run()
2024-04-29 01:41:53.032184670    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2024-04-29 01:41:53.032185925      self._target(*self._args, **self._kwargs)
2024-04-29 01:41:53.032187264    File "/opt/frigate/frigate/object_detection.py", line 102, in run_detector
2024-04-29 01:41:53.032188625      object_detector = LocalObjectDetector(detector_config=detector_config)
2024-04-29 01:41:53.032189903    File "/opt/frigate/frigate/object_detection.py", line 53, in __init__
2024-04-29 01:41:53.032191153      self.detect_api = create_detector(detector_config)
2024-04-29 01:41:53.032192551    File "/opt/frigate/frigate/detectors/__init__.py", line 18, in create_detector
2024-04-29 01:41:53.032193709      return api(detector_config)
2024-04-29 01:41:53.032208968    File "/opt/frigate/frigate/detectors/plugins/tensorrt.py", line 220, in __init__
2024-04-29 01:41:53.032210392      self.engine = self._load_engine(detector_config.model.path)
2024-04-29 01:41:53.032211768    File "/opt/frigate/frigate/detectors/plugins/tensorrt.py", line 88, in _load_engine
2024-04-29 01:41:53.032213111      with open(model_path, "rb") as f, trt.Runtime(self.trt_logger) as runtime:
2024-04-29 01:41:53.032214390  TypeError: expected str, bytes or os.PathLike object, not NoneType
2024-04-29 01:41:53.032215684  Exception ignored in: <function TensorRtDetector.__del__ at 0x7f8a3cc73dc0>
2024-04-29 01:41:53.032216892  Traceback (most recent call last):
2024-04-29 01:41:53.032218289    File "/opt/frigate/frigate/detectors/plugins/tensorrt.py", line 239, in __del__
2024-04-29 01:41:53.032219889  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::EfficientNMS_TRT version 1
2024-04-29 01:41:53.032237472      if self.outputs is not None:[2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::FlattenConcat_TRT version 1
2024-04-29 01:41:53.032238541  
2024-04-29 01:41:53.032239920  AttributeError: 'TensorRtDetector' object has no attribute 'outputs'
2024-04-29 01:41:53.032243322  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::fMHA_V2 version 1
2024-04-29 01:41:53.032260179  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::fMHCA version 1
2024-04-29 01:41:53.040063978  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::GenerateDetection_TRT version 1
2024-04-29 01:41:53.040080970  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::GridAnchor_TRT version 1
2024-04-29 01:41:53.040084963  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::GridAnchorRect_TRT version 1
2024-04-29 01:41:53.040088709  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::GroupNorm version 1
2024-04-29 01:41:53.040092522  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::InstanceNormalization_TRT version 1
2024-04-29 01:41:53.040096277  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::InstanceNormalization_TRT version 2
2024-04-29 01:41:53.040099915  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::LayerNorm version 1
2024-04-29 01:41:53.040103536  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::LReLU_TRT version 1
2024-04-29 01:41:53.040107166  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::MultilevelCropAndResize_TRT version 1
2024-04-29 01:41:53.040110754  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::MultilevelProposeROI_TRT version 1
2024-04-29 01:41:53.040114589  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::MultiscaleDeformableAttnPlugin_TRT version 1
2024-04-29 01:41:53.040118212  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::NMSDynamic_TRT version 1
2024-04-29 01:41:53.040121856  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::NMS_TRT version 1
2024-04-29 01:41:53.040125465  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::Normalize_TRT version 1
2024-04-29 01:41:53.040129145  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::PillarScatterPlugin version 1
2024-04-29 01:41:53.040132749  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::PriorBox_TRT version 1
2024-04-29 01:41:53.040136387  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::ProposalDynamic version 1
2024-04-29 01:41:53.040140197  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::ProposalLayer_TRT version 1
2024-04-29 01:41:53.040143769  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::Proposal version 1
2024-04-29 01:41:53.040147381  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::PyramidROIAlign_TRT version 1
2024-04-29 01:41:53.040151021  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::Region_TRT version 1
2024-04-29 01:41:53.040154628  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::Reorg_TRT version 1
2024-04-29 01:41:53.040158278  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::ResizeNearest_TRT version 1
2024-04-29 01:41:53.040161864  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::ROIAlign_TRT version 1
2024-04-29 01:41:53.040165566  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::RPROI_TRT version 1
2024-04-29 01:41:53.040355095  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::ScatterND version 1
2024-04-29 01:41:53.040359236  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::SeqLen2Spatial version 1
2024-04-29 01:41:53.040362920  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::SpecialSlice_TRT version 1
2024-04-29 01:41:53.040366586  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::SplitGeLU version 1
2024-04-29 01:41:53.040370158  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::Split version 1
2024-04-29 01:41:53.040373858  [2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::VoxelGeneratorPlugin version 1
2024-04-29 01:41:57.604314052  [2024-04-29 01:41:57] asyncio                        DEBUG   : Using selector: EpollSelector
2024-04-29 01:41:58.776831991  [2024-04-29 01:41:58] frigate.record.util            DEBUG   : End sync recordings.
2024-04-29 01:41:58.777054806  [2024-04-29 01:41:58] tzlocal                        DEBUG   : /etc/timezone found, contents:
2024-04-29 01:41:58.777056476   Etc/UTC
2024-04-29 01:41:58.777057763  
2024-04-29 01:41:58.777574984  [2024-04-29 01:41:58] tzlocal                        DEBUG   : /etc/localtime found
2024-04-29 01:41:58.778445862  [2024-04-29 01:41:58] tzlocal                        DEBUG   : 2 found:
2024-04-29 01:41:58.778447907   {'/etc/timezone': 'Etc/UTC', '/etc/localtime is a symlink to': 'Etc/UTC'}
2024-04-29 01:41:59.737835113  [INFO] Starting go2rtc healthcheck service...
2024-04-29 01:42:02.607133132  [2024-04-29 01:42:02] asyncio                        DEBUG   : Using selector: EpollSelector
2024-04-29 01:42:07.607207039  [2024-04-29 01:42:07] asyncio                        DEBUG   : Using selector: EpollSelector
2024-04-29 01:42:07.681540423  [2024-04-29 01:42:07] frigate.record.maintainer      DEBUG   : Copied /media/frigate/recordings/2024-04-29/00/kitchen/41.54.mp4 in 0.04902482032775879 seconds.
2024-04-29 01:42:07.717584132  [2024-04-29 01:42:07] frigate.record.maintainer      DEBUG   : Copied /media/frigate/recordings/2024-04-29/00/office/41.54.mp4 in 0.045224905014038086 seconds.
2024-04-29 01:42:12.607568196  [2024-04-29 01:42:12] asyncio                        DEBUG   : Using selector: EpollSelector
2024-04-29 01:42:12.789680066  [2024-04-29 01:42:12] frigate.watchdog               INFO    : Detection appears to have stopped. Exiting Frigate...
s6-rc: info: service legacy-services: stopping
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service nginx: stopping
s6-rc: info: service go2rtc-healthcheck: stopping
2024-04-29 01:42:12.814256917  [INFO] The go2rtc-healthcheck service exited with code 256 (by signal 15)
s6-rc: info: service go2rtc-healthcheck successfully stopped
2024-04-29 01:42:12.882401442  [INFO] Service NGINX exited with code 0 (by signal 0)
s6-rc: info: service nginx successfully stopped
s6-rc: info: service nginx-log: stopping
s6-rc: info: service frigate: stopping
2024-04-29 01:42:12.887513006  [2024-04-29 01:42:12] frigate.app                    INFO    : Stopping...
s6-rc: info: service nginx-log successfully stopped
2024-04-29 01:42:12.888921048  [2024-04-29 01:42:12] frigate.watchdog               INFO    : Exiting watchdog...
2024-04-29 01:42:12.889373091  [2024-04-29 01:42:12] frigate.storage                INFO    : Exiting storage maintainer...
2024-04-29 01:42:12.890709117  [2024-04-29 01:42:12] frigate.ptz.autotrack          INFO    : Exiting autotracker...
2024-04-29 01:42:12.890719068  [2024-04-29 01:42:12] frigate.events.cleanup         INFO    : Exiting event cleanup...
2024-04-29 01:42:12.890725171  [2024-04-29 01:42:12] frigate.record.cleanup         INFO    : Exiting recording cleanup...
2024-04-29 01:42:12.890994362  [2024-04-29 01:42:12] frigate.stats                  INFO    : Exiting stats emitter...
2024-04-29 01:42:13.008080257  [2024-04-29 01:42:13] frigate.object_processing      INFO    : Exiting object processor...
2024-04-29 01:42:13.130943265  [2024-04-29 01:42:13] frigate.comms.ws               INFO    : Exiting websocket client...
2024-04-29 01:42:13.799908858  [2024-04-29 01:42:13] frigate.events.maintainer      INFO    : Exiting event processor...
2024-04-29 01:42:13.804756180  [2024-04-29 01:42:13] frigate.record.maintainer      INFO    : Exiting recording maintenance...
2024-04-29 01:42:13.805986363  [2024-04-29 01:42:13] watchdog.office                INFO    : Terminating the existing ffmpeg process...
2024-04-29 01:42:13.806330008  [2024-04-29 01:42:13] watchdog.office                INFO    : Waiting for ffmpeg to exit gracefully...
2024-04-29 01:42:13.808532288  [2024-04-29 01:42:13] watchdog.kitchen               INFO    : Terminating the existing ffmpeg process...
2024-04-29 01:42:13.808698493  [2024-04-29 01:42:13] frigate.video                  INFO    : kitchen: exiting subprocess
2024-04-29 01:42:13.808701832  [2024-04-29 01:42:13] watchdog.kitchen               INFO    : Waiting for ffmpeg to exit gracefully...
2024-04-29 01:42:14.668247052  [2024-04-29 01:42:14] frigate.video                  INFO    : office: exiting subprocess
2024-04-29 01:42:15.224478948  [2024-04-29 01:42:15] frigate.output                 INFO    : exiting output process...
2024-04-29 01:42:16.561341474  /usr/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 9 leaked shared_memory objects to clean up at shutdown
2024-04-29 01:42:16.561345367    warnings.warn('resource_tracker: There appear to be %d '
2024-04-29 01:42:16.565202570  [INFO] Service Frigate exited with code 0 (by signal 0)
s6-rc: info: service frigate successfully stopped
s6-rc: info: service trt-model-prepare: stopping
s6-rc: info: service go2rtc: stopping
s6-rc: info: service frigate-log: stopping
2024-04-29 01:42:16.569248254  exit with signal: terminated
s6-rc: info: service trt-model-prepare successfully stopped
s6-rc: info: service frigate-log successfully stopped
2024-04-29 01:42:16.572336222  [INFO] The go2rtc service exited with code 0 (by signal 0)
s6-rc: info: service go2rtc successfully stopped
s6-rc: info: service go2rtc-log: stopping
s6-rc: info: service go2rtc-log successfully stopped
s6-rc: info: service log-prepare: stopping
s6-rc: info: service s6rc-fdholder: stopping
s6-rc: info: service log-prepare successfully stopped
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service s6rc-fdholder successfully stopped
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped

Operating system

Debian

Install method

Docker Compose

Coral version

Other

Any other information that may be helpful

Host is an old Asus ROG laptop

SMI output:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce GTX 1060        Off |   00000000:01:00.0 Off |                  N/A |
| N/A   65C    P2             27W /   78W |     334MiB /   6144MiB |      1%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A   1980673      C   ffmpeg                                        166MiB |
|    0   N/A  N/A   1980676      C   ffmpeg                                        166MiB |
+-----------------------------------------------------------------------------------------+

Answered by JustJarethB

Apr 29, 2024

Slightly embarrasing solution:
model (and it's prop path) is required in config.yml despite being tagged optional/automatic. Although I tested this, I incorrectly modified someone's example - fixing path typo has solved issue

View full answer

NickM-27 · 2024-04-29T01:20:44Z

NickM-27
Apr 29, 2024
Collaborator Sponsor

this might be one of the issues that is fixed in dev

6 replies

NickM-27 Apr 29, 2024
Collaborator Sponsor

you can run dev images from https://github.com/users/blakeblackshear/packages/container/package/frigate but keep in mind there have been significant changes that are not represented in the current documentation, if you don't have an existing frigate setup then might be the best way to get going though

JustJarethB Apr 29, 2024
Author

@NickM-27 I deployed most recent tensor dev tag by changing my docker compose to

    image: ghcr.io/blakeblackshear/frigate:dev-c9246bd-tensorrt

At first I got a deployment failure but running the same thing again spun up as normal, but I get the same error as mentioned above

JustJarethB Apr 29, 2024
Author

It looks as though the issue is
if self.outputs is not None:[2024-04-29 01:41:53] frigate.detectors.plugins.tensorrt
but I am not a python dev, and on inspecting File "/opt/frigate/frigate/detectors/plugins/tensorrt.py", line 239, I couldn't find an obvious error source

NickM-27 Apr 29, 2024
Collaborator Sponsor

this indicates a driver or compute level problem. Some users have seen that the compute level fix resolved it, but it is simply that the tensorrt does not initialize properly

JustJarethB Apr 29, 2024
Author

Are you able to point to any debugging methods I can take? GPUs and drivers are a bit above my head and I don't really know what's wrong, and searching the error doesn't seem to have relevant information

JustJarethB · 2024-04-29T22:35:39Z

JustJarethB
Apr 29, 2024
Author

Slightly embarrasing solution:
model (and it's prop path) is required in config.yml despite being tagged optional/automatic. Although I tested this, I incorrectly modified someone's example - fixing path typo has solved issue

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Detector Support]: TensorRT Crashing into Boot Loop #11144

{{title}}

Replies: 2 comments 6 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

[Detector Support]: TensorRT Crashing into Boot Loop #11144

JustJarethB Apr 29, 2024

Describe the problem you are having

Preface

Issue

Checklist

Version

Frigate config file

docker-compose file or Docker CLI command

Relevant log output

Operating system

Install method

Coral version

Any other information that may be helpful

Replies: 2 comments · 6 replies

NickM-27 Apr 29, 2024 Collaborator Sponsor

NickM-27 Apr 29, 2024 Collaborator Sponsor

JustJarethB Apr 29, 2024 Author

JustJarethB Apr 29, 2024 Author

NickM-27 Apr 29, 2024 Collaborator Sponsor

JustJarethB Apr 29, 2024 Author

JustJarethB Apr 29, 2024 Author

JustJarethB
Apr 29, 2024

Replies: 2 comments 6 replies

NickM-27
Apr 29, 2024
Collaborator Sponsor

NickM-27 Apr 29, 2024
Collaborator Sponsor

JustJarethB Apr 29, 2024
Author

JustJarethB Apr 29, 2024
Author

NickM-27 Apr 29, 2024
Collaborator Sponsor

JustJarethB Apr 29, 2024
Author

JustJarethB
Apr 29, 2024
Author