[Detector Support]: No EdgeTPU was detected on Proxmox LXC based setup

[jbolla]

Describe the problem you are having

I'm running frigate in a docker container which is running in an LXC container on proxmox. Whenever frigate tries to use a detector I get logs saying "No EdgeTPU was detected. If you do not have a Coral device yet, you must configure CPU detectors." and then the container restarts.

Version

0.13.1-34FB1C2

Frigate config file

ffmpeg:
  hwaccel_args: preset-vaapi

detectors:
  coral:
    type: edgetpu
    device: usb

#Global Object Settings
objects:
  track:
    - person
  filters:
    person:
      min_area: 5000
      max_area: 100000

go2rtc:
  streams:
    doorbell:
...

cameras:
  doorbell:
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        - path: rtsp://127.0.0.1:8554/doorbell
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/doorbell_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
            - audio
      hwaccel_args: preset-vaapi
    detect:
      enabled: True
    record:
      enabled: True
    snapshots:
      enabled: True

docker-compose file or Docker CLI command

version: '3.9'

services:

  frigate:
    container_name: frigate
    privileged: true
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:stable
    shm_size: "128mb" # update for your cameras based on calculation above
    devices:
      - /dev/bus/usb:/dev/bus/usb # for coral usb
      - /dev/coral_usb:/dev/coral_usb # coral usb pass through
      - /dev/dri/renderD128 # for intel hwaccel, needs to be updated for your hardware
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /opt/frigate/config:/config
      - /mnt/security_camera:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "5000:5000"
      - "1935:1935" # RTMP feeds
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
    environment:
      FRIGATE_RTSP_PASSWORD: "foobar"

Relevant log output

frigate.detectors.plugins.edgetpu_tfl ERROR   : No EdgeTPU was detected. If you do not have a Coral device yet, you must configure CPU detectors.

Operating system

Proxmox

Install method

Docker Compose

Coral version

USB

Any other information that may be helpful

On the proxmox host I can see the coral (sometimes as Global Unichip Corp. and sometimes in its Google, Inc.):

root@proxmox1:~# lsusb
Bus 002 Device 004: ID 18d1:9302 Google Inc. 
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 8087:0026 Intel Corp. AX201 Bluetooth
Bus 001 Device 002: ID 10c4:ea60 Silicon Labs CP210x UART Bridge
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
root@proxmox1:~# ls -la /dev/bus/usb/002/004
crw-rw-r-- 1 root root 189, 130 Feb 20 21:26 /dev/bus/usb/002/004

I've tried using udev to change some configuration and to assign it a static alias since it changes which usb device number it's listed as. udev is new territory for me, so I don't fully understand what I'm doing here:

root@proxmox1:~# cat /etc/udev/rules.d/98-coral.rules
SUBSYSTEMS=="usb", ATTRS{idVendor}=="1a6e", ATTRS{idProduct}=="089a", MODE="0664", TAG+="uaccess", SYMLINK+="coral_usb"
SUBSYSTEMS=="usb", ATTRS{idVendor}=="18d1", ATTRS{idProduct}=="9302", MODE="0664", TAG+="uaccess", SYMLINK+="coral_usb"
root@proxmox1:~# ls -la /dev/coral_usb 
lrwxrwxrwx 1 root root 15 Feb 20 21:26 /dev/coral_usb -> bus/usb/002/004

LXC container config:

arch: amd64
cores: 2
features: nesting=1
hostname: docker-frigate
memory: 8192
nameserver: 192.168.1.1
net0: name=eth0,bridge=vmbr0,gw=192.168.1.1,hwaddr=BC:24:11:1A:6F:58,ip=192.168.1.31/24,type=veth
onboot: 1
ostype: debian
rootfs: local-lvm:vm-103-disk-0,size=4G
swap: 512
tags: proxmox-helper-scripts
lxc.cgroup2.devices.allow: a
lxc.cap.drop:
lxc.cgroup2.devices.allow: c 188:* rwm
lxc.cgroup2.devices.allow: c 189:* rwm
lxc.mount.entry: /dev/serial/by-id  dev/serial/by-id  none bind,optional,create=dir
lxc.mount.entry: /dev/ttyUSB0       dev/ttyUSB0       none bind,optional,create=file
lxc.mount.entry: /dev/ttyUSB1       dev/ttyUSB1       none bind,optional,create=file
lxc.mount.entry: /dev/ttyACM0       dev/ttyACM0       none bind,optional,create=file
lxc.mount.entry: /dev/ttyACM1       dev/ttyACM1       none bind,optional,create=file
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file 0, 0
lxc.mount.entry: /dev/bus/usb/001/004 dev/bus/usb/001/004 none bind,optional,create=file 0, 0
lxc.mount.entry: /dev/bus/usb/001/002 dev/bus/usb/001/002 none bind,optional,create=file 0, 0
lxc.mount.entry: /dev/bus/usb/002/003 dev/bus/usb/002/003 none bind,optional,create=file 0, 0
lxc.mount.entry: /dev/coral_usb     dev/coral_usb     none bind,optional,create=file 0, 0
lxc.mount.auto: cgroup:rw

Within the LXC container I can see the coral usb device:

root@docker-frigate:~# lsusb
Bus 002 Device 003: ID 1a6e:089a Global Unichip Corp. 
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 8087:0026 Intel Corp. AX201 Bluetooth
Bus 001 Device 002: ID 10c4:ea60 Silicon Labs CP210x UART Bridge
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
root@docker-frigate:~# ls -la /dev/bus/usb/00
001/ 002/ 
root@docker-frigate:~# ls -la /dev/bus/usb/002/003 
crw-rw-r-- 1 root root 189, 130 Feb 20 21:26 /dev/bus/usb/002/003
root@docker-frigate:~# ls -la /dev/coral_usb 
crw-rw-r-- 1 root root 189, 130 Feb 20 21:26 /dev/coral_usb

Within the LXC container I'm starting up the frigate docker container with the included docker compose configuration.

Within the docker container I can see the coral usb device as well.

root@063832abd744:/opt/frigate# lsusb 
Bus 002 Device 004: ID 18d1:9302  
Bus 002 Device 001: ID 1d6b:0003 Linux 6.5.11-8-pve xhci-hcd xHCI Host Controller
Bus 001 Device 002: ID 10c4:ea60 Nabu Casa SkyConnect v1.0
Bus 001 Device 003: ID 8087:0026  
Bus 001 Device 001: ID 1d6b:0002 Linux 6.5.11-8-pve xhci-hcd xHCI Host Controlle

In the docker container I don't see /dev/bus/usb/002/004 in the filesystem but I do see dev/coral_usb

root@063832abd744:/opt/frigate# ls -la /dev/bus/usb/002/
total 0
drwxr-xr-x 2 root root       60 Feb 21 12:40 .
drwxr-xr-x 4 root root       80 Feb 21 12:40 ..
c--------- 1 root root 189, 130 Feb 21 12:40 003
root@063832abd744:/opt/frigate# ls -la /dev/coral_usb 
c--------- 1 root root 189, 130 Feb 21 12:40 /dev/coral_usb

The coral always has the same device ID in all three of the proxmox host, the LCX container, and the docker container. I'm not sure how to trigger it to change from one to the other, but I've confirmed that frigate doesn't work in either case. I've also tried running coral python inference code in the docker container. It fails with "Failed to load delegate from libedgetpu.so.1.0 with tflite_runtime" which I understand means its not finding the coral either. I tried to run the same python coral project in the LXC container but my python version there there is too recent and I gave up after spending too long struggling with trying to get a venv or third party wheel to work.

If I comment out the detector section from my frigate config it starts up fine and I can see my camera feed.

[NickM-27]

this is going to be a proxmox config issue

see #9019 (comment) for a similar case

[jbolla]

You were right the problem was proxmox config. I did not have the correct usb device mounted. The underlying issue was that the device number of the coral is not stable, so my config which passed in /dev/bus/002/003 became incorrect when the coral changed to /dev/bus/002/004. I think that the bus is stable, so I've passed through /dev/bus/002/001 ... /dev/bus/002/004 and now frigate can find it.

This also means that my attempts to use udev didn't work. The symlink is showing up correctly on the proxmox host /dev/coral_usb -> bus/usb/002/004, and it shows up in the containers, but it appears that doesn't actually make the coral available for use.

Finally, a fact that surprised me, running lsusb in a container is able to list usb devices connected to the hardware even when they are not properly mounted to the container, so you cannot treat presence in the list as evidence you are passing it through correctly.

[marcinsu]

What configuration did you do to fix this problem?
I'm currently using 2x Coral USB (proxmox -> LXC -> docker) and I see that every now and then Frigate loses access to one or both devices.

[jbolla]

Ultimately I got it working consistently by passing in all of /dev/bus/###/###.