[Detector Support]: ONNX detector not using Jetson's GPU

[marmbrus]

Describe the problem you are having

I'm trying to run Frigate+ models using the new onnx detector on my NVidia Jetson AGX Orin developer kit. While it seems to be working, the CPU is over 1000% and the GPU is at 0%.

I've been able to successfully use the GPU with the tensort detector type.

Version

0.15.0-33825f6

Frigate config file

mqtt:
  enabled: false

auth:
  enabled: false

tls:
  enabled: false

cameras:
  frontpatio:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://admin:####@frontpatio.cams/Streaming/Channels/1
          roles:
            - record
            - audio
            - detect
    audio:
      enabled: true

    zones:
      inside:
        coordinates:
          0,0.648,0.464,0.212,0.507,0.218,0.503,0.383,0.605,0.412,0.613,0.328,0.744,0.453,0.843,0.543,0.795,0.73,0.859,1,0,1
        loitering_time: 0
        inertia: 3
      door:
        coordinates: 0.814,0.732,0.897,0.287,0.984,0.354,0.914,0.619,0.844,0.88
        loitering_time: 0
      neighbor:
        coordinates: 0.006,0.637,0.462,0.204,0.655,0.245,0.699,0,0,0
        loitering_time: 0

        inertia: 3
      outside:
        coordinates: 0.519,0.218,0.51,0.369,0.596,0.414,0.625,0.24
        loitering_time: 0
    motion:
      mask: 0.022,0.099,0.264,0.095,0.266,0.05,0.02,0.049
    objects:
      mask: 0.654,0,0.606,0.287,0.643,0.39,0.666,0.474,0.794,0.483,1,0.727,0.998,0.005
      filters:
        person:
          mask:
            - 0.654,0,0.606,0.287,0.643,0.39,0.666,0.474,0.794,0.483,1,0.727,0.998,0.005
            - 0.749,0.692,0.803,0.72,0.795,0.768,0.743,0.744
  frontwalkway:
    ffmpeg:
      inputs:
        - path: rtsp://admin:####@frontwalkway.cams/h264Preview_01_main
          roles:
            - record
            - audio
            - detect

    zones:
      neighbor:
        coordinates: 0,0.477,0.227,0.18,0.744,0.194,0.761,0.003,-0.001,0.005
        loitering_time: 0
        inertia: 3
      inside:
        coordinates:
          -0.001,0.489,0.23,0.194,0.626,0.209,0.74,0.319,0.77,0.011,0.998,0.005,0.998,0.998,0,0.995
        loitering_time: 0
        inertia: 3
      outside:
        coordinates: 0.659,0.196,0.724,0.266,0.74,0.196
        loitering_time: 0
        inertia: 3
  cats:
    ffmpeg:
      inputs:
        - path: rtsp://admin:####@cats.cams/Streaming/Channels/1
          roles:
            - record
            - audio
        - path: rtsp://admin:####@cats.cams/Streaming/Channels/2
          roles:
            - detect
    motion:
      mask: 0.023,0.09,0.269,0.088,0.268,0.042,0.018,0.031
    zones:
      inside:
        coordinates: 0.257,0.477,0.234,0.993,0.495,1,0.465,0.456
        loitering_time: 0

record:
  enabled: true
  sync_recordings: false
  retain:
    days: 30
    mode: motion

snapshots:
  enabled: true
  clean_copy: true
  bounding_box: true
  crop: true
  retain:
    default: 365

ffmpeg:
  hwaccel_args: preset-jetson-h264

detectors:
  onnx:
    type: onnx

model:
  path: plus://5cbbe20f180851aab60ab8e498c8516c

objects:
  track:
    - person
    - cat
    - dog
    - deer
    - car
    - package

version: 0.15-0

docker-compose file or Docker CLI command

services:
  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    runtime: nvidia
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:0.15.0-beta1-tensorrt-jp5
    shm_size: "5000mb" # update for your cameras based on calculation above

    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /home/marmbrus/frigate/config:/config
      - frigate-media-new:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "8971:8971"
      - "5000:5000" # Internal unauthenticated access. Expose carefully.
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
    environment:
      - PLUS_API_KEY=XXXX

volumes:
  frigate-media-new:
    driver_opts:
      type: nfs
      o: "addr=memoryalpha.cams"
      device: ":/volume1/frigate2"

Relevant Frigate log output

marmbrus@jetson2:~/frigate$ sudo docker compose logs -f | grep onnx
frigate  | 2024-10-29 09:50:09.952368235  [2024-10-29 09:50:09] detector.onnx                  INFO    : Starting detection process: 438
frigate  | 2024-10-29 09:50:10.001473115  [2024-10-29 09:50:09] frigate.detectors.plugins.onnx INFO    : ONNX: loaded onnxruntime module
frigate  | 2024-10-29 09:50:10.001513212  [2024-10-29 09:50:09] frigate.detectors.plugins.onnx INFO    : ONNX: loading /config/model_cache/5cbbe20f180851aab60ab8e498c8516c
frigate  | 2024-10-29 09:50:10.043760839  [2024-10-29 09:50:10] frigate.detectors.plugins.onnx INFO    : ONNX: /config/model_cache/5cbbe20f180851aab60ab8e498c8516c loaded

Install method

Docker Compose

Object Detector

Other

Screenshots of the Frigate UI's System metrics pages

Any other information that may be helpful

No response

[NickM-27]

Jetson is not supported with Frigate+ as the jetson builds do not have jetson specific onnxruntime installed. There is no active maintainer for the jetson builds so there is no way to really test that it works. You would have to build the onnxruntime wheel and then install it inside of the Frigate container https://onnxruntime.ai/docs/build/eps.html#nvidia-jetson-tx1tx2nanoxavierorin

[marmbrus]

Thank you for the super fast response!

For anyone else who runs into this, I was able to use the GPU with the following changes to the container:

FROM ghcr.io/blakeblackshear/frigate:0.15.0-beta1-tensorrt-jp5

RUN pip uninstall -y onnxruntime

# Download the wheel file
ADD https://nvidia.box.com/shared/static/9aemm4grzbbkfaesg5l7fplgjtmswhj8.whl /tmp/onnxruntime_gpu-1.15.1-cp39-cp39-linux_aarch64.whl

# Install the wheel using pip
RUN pip install --no-deps /tmp/onnxruntime_gpu-1.15.1-cp39-cp39-linux_aarch64.whl

RUN rm /tmp/onnxruntime_gpu-1.15.1-cp39-cp39-linux_aarch64.whl

[NickM-27]

where did you get that link? it could potentially be added to the docker install

[marmbrus]

Link is from here: https://elinux.org/Jetson_Zoo#ONNX_Runtime

Would be awesome to get this added automatically. If you have a pointer and the approach above seems reasonable I'd be willing to try adding it.