OpenVino On AMD Ryzen CPU

[adarshahd]

Hello all,

First of all, thank you very much for a great project. I have a MinisForum UM790 pro (Ryzen 7945HS) mini pc, which has Radeon 780M gpu. Unfortunately I had to use CPU detectors as AMD doesn't have a proper SW for using GPU for this use case, although this chipset has inbuilt TPU (8 TOPS). So my inference speeds were anywhere around 30ms and was not happy

So today I found out we can use OpenVINO detectors for AMD CPU/GPUs also. And alas ! my inference speeds has dropped to 3ms !!

This is FYI all

[NickM-27]

what is your config?

[adarshahd]

Here it is, pretty basic. Just that using OpenVINO instead of CPU

mqtt:
  enabled: false
ffmpeg:
  hwaccel_args: preset-vaapi
  output_args:
    record: preset-record-generic-audio-aac
cameras:
  front:
    ffmpeg:
      inputs:
        - path: rtsp://***:***@192.168.1.5:554/stream1
          roles:
            - detect
    detect:
      width: 1920
      height: 1080
record:
  enabled: True
  retain:
    days: 7
    mode: motion
  events:
    retain:
      default: 14
      mode: active_objects
detectors:
  ov:
    type: openvino
    device: AUTO
    model:
      path: /openvino-model/ssdlite_mobilenet_v2.xml
model:
  width: 300
  height: 300
  input_tensor: nhwc
  input_pixel_format: bgr
  labelmap_path: /openvino-model/coco_91cl_bkgr.txt

[NickM-27]

If I had to guess it is still using the CPU (not the GPU), the only way to confirm for sure that it is using the GPU will be to change the device from AUTO to GPU

[adarshahd]

You are right. It would still use the CPU. We can see from the log that it fails if we change the config to use GPU.

RuntimeError: Failed to create plugin /usr/local/lib/python3.9/dist-packages/openvino/libs/libopenvino_intel_gpu_plugin.so for device GPU

However even with CPU the performance gained is enormous from 30ms -> 3ms. I can also see CPU usage going down from previous values

[adarshahd]

I also updated the title accordingly :)

[adarshahd]

And docker-compose. I run Fedora and use podman

version: "3.9"
services:
  frigate:
    container_name: frigate
    restart: unless-stopped
#    privileged: true
    image: ghcr.io/blakeblackshear/frigate:0.13.0-beta6
    shm_size: "64mb" # update for your cameras based on calculation above
    volumes:
      - ./config/:/config/:Z
      - ./storage:/media/frigate:Z
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    devices:
      - /dev/dri/renderD128:/dev/dri/renderD128
    ports:
      - "5000:5000"
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
    environment:
      FRIGATE_RTSP_PASSWORD: "password"
      NVIDIA_VISIBLE_DEVICES: void
      LIBVA_DRIVER_NAME: radeonsi

[GrumpyMeow]

I'm curious why the performance would improve using OpenVINO on a AMD Ryzen CPU.
Is it because of the model which is more efficient?
Is the "detector" more efficient?
Or is it using more efficient CPU features?

[NickM-27]

it is because the openvino detector is more efficient than running tflite directly on the cpu

[marcschumacher]

This might be a very basic question, but I read that OpenVINO was mostly created for Intel CPUs. Can you tell me how to install the drivers or is there any documentation available? Furthermore, can I assume that using the hardware acceleration itself also helps for decoding the ffmpeg streams?
I am currently using my MinisForum UM790 with Proxmox, including a Coral USB device, but I am having a lot of cameras and I would like to benefit from any additional performance my machine provides.

[ivanjx]

which intel cpu you were using before switching to the amd cpu? also what is the cpu usage while detecting an object on the amd cpu? thanks

[vishalrao88]

Thanks for sharing!

I was able to run openvino on a AMD 5800U (15W) with success! yolov8n: PT with ROCM -> 35-45 ms vs Openvino (amd cpu only) -> 25-35 ms.

I am interested in further optimizations. Have you had success with Rocm or other combos? e.g. RocM+Openvino. Anything thats faster than openvino (cpu only)?

[ivanjx]

what is the cpu usage currently?

[vishalrao88]

iirc about 70-75% while the code's running