[Detector Support]: frigate.detector.coral command uses high CPU

[greberg]

Describe the problem you are having

Can't get my head around the CPU usage. Using htop I see the command frigate.detector.coral uses a very high amount of CPU. Running approx somewhere in between 40-55% CPU. Not really sure how this is working towards the TPU Coral that I have. Lately I have seen some strange behaviour in the CPU usage as well. Staring up it looks nice and stable for a while but after a while it get some peaks, see picture

Also my htop

System page Frigate:

Version

0.13.2

Frigate config file

mqtt:
  host: x
  user: x
  password: x
detectors:
  coral:
    type: edgetpu
    device: usb
ffmpeg:
  #hwaccel_args: preset-intel-qsv-h264
  hwaccel_args: preset-vaapi
  #hwaccel_args: preset-intel-qsv-h265
objects:
  # Optional: list of objects to track from labelmap.txt (default: shown below)
  track:
    - person
    - car
    - dog
    - cat
  filters:
    person:
      min_area: 7000
      max_area: 150000
      threshold: 0.75 # min decimal percentage for tracked object's computed score to be considered a true positive (default: 0.7)
    car:
      threshold: 0.75
# Optional: Configuration for the jpg snapshots written to the clips directory for each event
# NOTE: Can be overridden at the camera level
snapshots:
  # Optional: Enable writing jpg snapshot to /media/frigate/clips (default: shown below)
  enabled: True
  # Optional: save a clean PNG copy of the snapshot image (default: shown below)
  clean_copy: True
  # Optional: print a timestamp on the snapshots (default: shown below)
  timestamp: True
  # Optional: draw bounding box on the snapshots (default: shown below)
  bounding_box: True
  # Optional: crop the snapshot (default: shown below)
  crop: False
  # Optional: height to resize the snapshot to (default: original size)
  #height: 175
  # Optional: Restrict snapshots to objects that entered any of the listed zones (default: no required zones)
  required_zones: []
  # Optional: Camera override for retention settings (default: global values)
  retain:
    # Required: Default retention days (default: shown below)
    default: 5
    # Optional: Per object retention days
    objects:
      person: 7
rtmp:
  # Optional: Enable the RTMP stream (default: False)
  enabled: False
go2rtc:
  streams:
    garage:
      - rtspx://10.0.1.135:7441/MReZEs2SIebHXUxz
      #- rtspx://10.0.1.135:7441/yPSmxlWYocvIyI3e
    baksidapool:
      - rtspx://10.0.1.135:7441/PyaoZ7aGGZNtZfxB
    entre:
      - rtspx://10.0.1.135:7441/3amd5wRpOoReXCR5
    baksidaingang:
      - rtspx://10.0.1.135:7441/tJXjkKMSzq3M19Yp
    baksidapoolhus:
      - rtspx://10.0.1.135:7441/pYedtbkD8F0kLdo1
    baksidahus:
      - rtspx://10.0.1.135:7441/zxNLFE1mUSWCwOtl
    framsidahus:
      - rtspx://10.0.1.135:7441/nVCaqTFLsdbIHmwq


cameras:
  garage: # <------ Name the camera
    ffmpeg:
      inputs:
        #- path: rtsps://10.0.1.135:7441/yPSmxlWYocvIyI3e?enableSrtp
        - path: rtsps://10.0.1.135:7441/MReZEs2SIebHXUxz
          roles:
            - detect
      output_args:
        record: preset-record-ubiquiti
    detect:
      enabled: True
      fps: 5
      #width: 2688 # <---- update for your camera's resolution
      #height: 1512 # <---- update for your camera's resolution
      width: 1280 # <---- update for your camera's resolution
      height: 720 # <---- update for your camera's resolution

      stationary:
        interval: 10
        threshold: 50
    motion:
      mask:
        - 0,0,218,0,219,64,0,67
    zones:
      garageplatta:
        coordinates: 1098,720,1280,348,627,77,0,294,0,720
      outside:
        coordinates: 1280,0,1280,342,606,59,602,0
  baksidapool: # <------ Name the camera
    objects:
      # Optional: list of objects to track from labelmap.txt (default: shown below)
      track:
        - person
        - dog
        - cat
    ffmpeg:
      #hwaccel_args: preset-vaapi
      #output_args:
      #  record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -ar 44100 -c:a aac
       # rtmp: -c:v copy -f flv -ar 44100 -c:a aac
      inputs:
        #- path: rtsps://10.0.1.135:7441/YW7kPHYEy6Rq80p0?enableSrtp
        - path: rtsps://10.0.1.135:7441/PyaoZ7aGGZNtZfxB
          roles:
            - detect
      output_args:
        record: preset-record-ubiquiti
    detect:
      enabled: True
      fps: 5
      #width: 1920 # <---- update for your camera's resolution
      #height: 1080 # <---- update for your camera's resolution
      width: 1024 # <---- update for your camera's resolution
      height: 576 # <---- update for your camera's resolution
  entre: # <------ Name the camera
      # Optional: list of objects to track from labelmap.txt (default: shown below)
    ffmpeg:
      inputs:
        #- path: rtsps://10.0.1.135:7441/9emTfdSN66BoZGaB?enableSrtp
        - path: rtsps://10.0.1.135:7441/3amd5wRpOoReXCR5
          roles:
            - detect
      output_args:
        record: preset-record-ubiquiti
    detect:
      enabled: True
      fps: 5
      #width: 2688 # <---- update for your camera's resolution
      #height: 1512 # <---- update for your camera's resolution
      width: 1280 # <---- update for your camera's resolution
      height: 720 # <---- update for your camera's resolution
    motion:
      mask:
        - 0,0,0,55,64,53,66,28,212,30,219,0
    zones:
      entrance: # Hela framsidan vid entren
        coordinates: 765,720,1214,121,936,26,0,321,0,720,442,720
          #2688,291,2419,1489,242,1512,0,1512,0,666,1993,67
    objects:
      filters:
        #car:
        #  mask:
        #    - 947,1512,2554,246,1982,31,239,560
        #cat:
        #  mask:
        #    - 0,1512,2382,1512,2688,267,1964,23,0,619
        #dog:
        #  mask:
        #    - 0,1512,2390,1512,2688,272,2688,267,0,608
        person:
          mask:
            - 360,177,712,382,1147,96,1217,118,763,720,449,720,112,268
  baksidaingang: # <------ Name the camera
    objects:
      # Optional: list of objects to track from labelmap.txt (default: shown below)
      track:
        - person
        - dog
        - cat
    ffmpeg:

      inputs:
        #- path: rtsps://10.0.1.135:7441/EfGqIY8nSGvdFexj?enableSrtp
        - path: rtsps://10.0.1.135:7441/tJXjkKMSzq3M19Yp
          roles:
            - detect
      output_args:
        record: preset-record-ubiquiti
    detect:
      enabled: True
      fps: 5
      #width: 1920 # <---- update for your camera's resolution
      #height: 1080 # <---- update for your camera's resolution
      width: 1024 # <---- update for your camera's resolution
      height: 576 # <---- update for your camera's resolution
  baksidapoolhus: # <------ Name the camera
    objects:
      # Optional: list of objects to track from labelmap.txt (default: shown below)
      track:
        - person
        - dog
        - cat
    ffmpeg:
      inputs:
      #- path: rtsps://10.0.1.135:7441/l5FGJpZO4DiSbJUe?enableSrtp
      - path: rtsps://10.0.1.135:7441/pYedtbkD8F0kLdo1
        roles:
          - detect
      output_args:
        record: preset-record-ubiquiti
    detect:
      enabled: True
      fps: 5
      #width: 1920 # <---- update for your camera's resolution
      #height: 1080 # <---- update for your camera's resolution
      width: 1024 # <---- update for your camera's resolution
      height: 576 # <---- update for your camera's resolution

  baksidahus: # <------ Name the camera
    objects:
      # Optional: list of objects to track from labelmap.txt (default: shown below)
      track:
        - person
        - dog
        - cat
    ffmpeg:
      inputs:
        #- path: rtsps://10.0.1.135:7441/H1BBXa3eXmwxNr4G?enableSrtp
        - path: rtsps://10.0.1.135:7441/zxNLFE1mUSWCwOtl
          roles:
            - detect
      output_args:
        record: preset-record-ubiquiti
    detect:
      enabled: True
      fps: 5
      #width: 1920 # <---- update for your camera's resolution
      #height: 1080 # <---- update for your camera's resolution
      width: 1024 # <---- update for your camera's resolution
      height: 576 # <---- update for your camera's resolution
  framsidahus: # <------ Name the camera
    objects:
      # Optional: list of objects to track from labelmap.txt (default: shown below)
      track:
        - person
        - dog
        - cat
    ffmpeg:
      inputs:
        #- path: rtsps://10.0.1.135:7441/PvRHc1ZJQ3XTCRmz?enableSrtp
        - path: rtsps://10.0.1.135:7441/nVCaqTFLsdbIHmwq
          roles:
            - detect
      output_args:
        record: preset-record-ubiquiti
    detect:
      enabled: True
      fps: 5
      #width: 1920 # <---- update for your camera's resolution
      #height: 1080 # <---- update for your camera's resolution
      width: 1024 # <---- update for your camera's resolution
      height: 576 # <---- update for your camera's resolution

docker-compose file or Docker CLI command

rigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:0.13.2
    #image: ghcr.io/blakeblackshear/frigate:0.12.0-rc2
    #image: ghcr.io/blakeblackshear/frigate:0.13.0-rc1
    shm_size: "360mb"
    devices:
      - /dev/bus/usb:/dev/bus/usb
      - /dev/dri/renderD128
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ../frigate:/config
      - ../frigate/storage:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    #environment:
    #  LIBVA_DRIVER_NAME: 'i965'
    ports:
      - "5001:5000"
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp

Relevant log output

2024-02-22 14:39:11.926607569  [INFO] Preparing Frigate...
2024-02-22 14:39:11.952350254  [INFO] Starting Frigate...
2024-02-22 14:39:13.270807711  [2024-02-22 14:39:13] frigate.app                    INFO    : Starting Frigate (0.13.2-6476f8a)
2024-02-22 14:39:13.345218270  [2024-02-22 14:39:13] peewee_migrate.logs            INFO    : Starting migrations
2024-02-22 14:39:13.348991819  [2024-02-22 14:39:13] peewee_migrate.logs            INFO    : There is nothing to migrate
2024-02-22 14:39:13.352725259  [2024-02-22 14:39:13] frigate.app                    INFO    : Recording process started: 297
2024-02-22 14:39:13.354853445  [2024-02-22 14:39:13] frigate.app                    INFO    : go2rtc process pid: 92
2024-02-22 14:39:13.380775820  [2024-02-22 14:39:13] detector.coral                 INFO    : Starting detection process: 307
2024-02-22 14:39:16.027126194  [2024-02-22 14:39:13] frigate.app                    INFO    : Output process started: 309
2024-02-22 14:39:16.027336721  [2024-02-22 14:39:13] frigate.app                    INFO    : Camera processor started for garage: 320
2024-02-22 14:39:16.027573509  [2024-02-22 14:39:13] frigate.app                    INFO    : Camera processor started for baksidapool: 322
2024-02-22 14:39:16.027650442  [2024-02-22 14:39:13] frigate.app                    INFO    : Camera processor started for entre: 323
2024-02-22 14:39:16.027722094  [2024-02-22 14:39:13] frigate.app                    INFO    : Camera processor started for baksidaingang: 327
2024-02-22 14:39:16.027792712  [2024-02-22 14:39:13] frigate.app                    INFO    : Camera processor started for baksidapoolhus: 328
2024-02-22 14:39:16.027862905  [2024-02-22 14:39:13] frigate.app                    INFO    : Camera processor started for baksidahus: 329
2024-02-22 14:39:16.027976850  [2024-02-22 14:39:13] frigate.app                    INFO    : Camera processor started for framsidahus: 333
2024-02-22 14:39:16.042834101  [2024-02-22 14:39:13] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2024-02-22 14:39:16.047592869  [2024-02-22 14:39:13] frigate.app                    INFO    : Capture process started for garage: 336
2024-02-22 14:39:16.047703155  [2024-02-22 14:39:16] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2024-02-22 14:39:16.047802912  [2024-02-22 14:39:13] frigate.app                    INFO    : Capture process started for baksidapool: 339
2024-02-22 14:39:16.048956669  [2024-02-22 14:39:13] frigate.app                    INFO    : Capture process started for entre: 344
2024-02-22 14:39:16.048961277  [2024-02-22 14:39:13] frigate.app                    INFO    : Capture process started for baksidaingang: 354
2024-02-22 14:39:16.049063739  [2024-02-22 14:39:13] frigate.app                    INFO    : Capture process started for baksidapoolhus: 363
2024-02-22 14:39:16.049165733  [2024-02-22 14:39:13] frigate.app                    INFO    : Capture process started for baksidahus: 370
2024-02-22 14:39:16.049268475  [2024-02-22 14:39:13] frigate.app                    INFO    : Capture process started for framsidahus: 376

Operating system

Other

Install method

Docker Compose

Coral version

USB

Any other information that may be helpful

No response

[NickM-27]

can we see a full screenshot of the system page while this is occurring

[greberg]

All of a sudden it started to appear again. Something makes the CPU go bananas.

Logs also reports tone of errors

And htop shows a lot of frigate threads

I have tried to restart the frigate container and rebooted the server as well to see if something was locked but nothing help.

[LukeHandle]

To add another datapoint, I'm also seeing very high frigate.detector.coral usage on 1 thread. I suspect this is due to a lack of Motion Masks on 1 of my cameras triggering many calls to the coral TPU.

It'd be interesting to have these sorts of statistics presented in the UI - eg. camera X is sending Y events per second/minute to the configured detector?

[NickM-27]

It'd be interesting to have these sorts of statistics presented in the UI - eg. camera X is sending Y events per second/minute to the configured detector?

it already is shown in the UI, on the system page

[LukeHandle]

Ahhh, you're referring to the FPS that is on the Detect thread? I guess the leap I hadn't made was that the detected FPS here was what then sent to Coral detector process? Based on CPU, frigate.detector.coral must be doing some processing before sending to the TPU (otherwise I'd expect smaller CPU as the processing occurred on TPU?)?

Per frame of the camera feed, with multiple (red) motion boxes, does that still send just 1 image to the detector thread, I wonder if many smaller motion boxes (5-10 or so I think I saw) per frame could cause higher CPU vs. a single red box per frame?

I see this for coral detector:

P-ID	Inference Speed	CPU %	Memory %
306	61.15 ms	39.4%	1.7%

And this for a single camera:

Process	P-ID	FPS	CPU %	Memory %
ffmpeg	351	5.1	5.8%	0.7%
Capture	344	5	2.0%	1.5%
Detect	332	4.9 (0 skipped)	12.9%	1.7%

(on 0.13.2-6476F8A)

[NickM-27]

Yes, the coral process has to change the pixel format and crop. Motion boxes turn into regions based on joining logic. There is not one region for each motion box.

The first problem I see is your coral inference times is quite slow.