Process Frigate images with DeepStack, Facebox, CompreFace, and AWS Rekognition

[jakowenko]

Double Take

Unified UI and API for processing and training images for facial recognition.

Why?

There's a lot of great open source software to perform facial recognition, but each of them behave differently. Double Take was created to abstract the complexities of the detection services and combine them into an easy to use UI and API.

Features

• UI and API bundled into single Docker image
• Ability to password protect UI and API
• Support for multiple detectors
• Train and untrain images for subjects
• Process images from NVRs
• Publish results to MQTT topics
• REST API can be invoked by other applications

Supported Detectors

• DeepStack v2021.02.1-2021.06.01
• CompreFace v0.5.0-0.5.1
• Facebox

Supported NVRs

• Frigate v0.8.0-0.9.0

Integrations

Frigate

Subscribe to Frigate's MQTT topics and process images for analysis.

mqtt:
  host: 192.168.1.1

frigate:
  url: http://192.168.1.1:5000

When the frigate/events topic is updated the API begins to process the snapshot.jpg and latest.jpg images from Frigate's API. These images are passed from the API to the configured detector(s) until a match is found that meets the configured requirements. To improve the chances of finding a match, the processing of the images will repeat until the amount of retries is exhausted or a match is found.

When the frigate/+/person/snapshot topic is updated the API will process that image with the configured detector(s). It is recommended to increase the MQTT snapshot size in the Frigate camera config.

cameras:
  front-door:
    mqtt:
      timestamp: False
      bounding_box: False
      crop: True
      height: 500

If a match is found the image is saved to /.storage/matches/${filename}.

Home Assistant

Trigger automations / notifications when images are processed.

If the MQTT integration is configured within Home Assistant, then sensors will automatically be created.

Notification Automation

alias: Notify
trigger:
  - platform: state
    entity_id: sensor.double_take_david
  - platform: state
    entity_id: sensor.double_take_unknown
condition:
  - condition: template
    value_template: '{{ trigger.to_state.state != trigger.from_state.state }}'
action:
  - service: notify.mobile_app
    data:
      message: >-
        {{trigger.to_state.attributes.friendly_name}} is near the
        {{trigger.to_state.state}} @
        {{trigger.to_state.attributes.match.confidence}}% by
        {{trigger.to_state.attributes.match.detector}}:{{trigger.to_state.attributes.match.type}}
        taking {{trigger.to_state.attributes.attempts}} attempt(s) @
        {{trigger.to_state.attributes.duration}} sec
      data:
        attachment:
          url: http://192.168.1.2:3000/api/storage/matches/{{trigger.to_state.attributes.match.filename}}?box=true&token={{trigger.to_state.attributes.token}}
        actions:
          - action: URI
            title: View Image
            uri: http://192.168.1.2:3000/api/storage/matches/{{trigger.to_state.attributes.match.filename}}?box=true&token={{trigger.to_state.attributes.token}}

I was hoping for feedback from the community and wanted to see if others would find something like this useful.

Github: https://github.com/jakowenko/double-take
Docker Hub: https://hub.docker.com/r/jakowenko/double-take

[hawkeye217]

I do something similar but I just use HA automations and the Facebox integration (https://www.home-assistant.io/integrations/facebox/)

Looks like your solution could work for folks on other home automation platforms too.

[jakowenko]

How are you liking that integration? The main difference I see is that the Facebox integration relies on that camera entity being updated to perform the processing, right? So you would need to have an entity setup for each camera and then process the jpgs with the integration.

The way my current API works is that when a Frigate event is POSTed to it, it will process up to X number of snapshot.jpg images from Frigate for that event ID and also process the latest.jpg images from that camera feed. So it's looking at two images at once and looping that process until the threshold is met or a match is found.

[hawkeye217]

Yeah, your method as described works better than my existing setup. I just have a local_file camera entity that gets updated with the image from Frigate based on the mqtt message (using the camera.snapshot service), and then I call image_processing.detect_face on that generic camera entity.

[jakowenko]

Ya that makes sense. I wanted to process as many images as possible within reason when a new event comes in to better increase the chances of finding a matching face.

If you end up trying this out and run into any issues, let me know!

[hawkeye217]

Definitely. Thanks for the contribution!

[jakowenko]

I recently added support for subscribing and publishing to MQTT directly, instead of the REST API having to be invoked.

Also found out about the the CompreFace project and added support for their facial recognition. So far the results have been pretty promising!

README now includes all the configuration options that can be passed to the Docker container.

https://github.com/jakowenko/double-take

[jakowenko]

I've made quite a few updates over the past few weeks including support for DeepStack and added a lot more information in the README.

I'm going to work on supporting just face detection next since DeepStack supports that out of the box.

[jakowenko]

I've made quite a few UI updates over the past few weeks and the root of the API at http://YOUR-IP:PORT will now serve a frontend that shows all matches and misses from the detectors. You can either retrain the models with the images or delete them. More updates to follow!

[jakowenko]

I pushed quite a few updates and small performance improvements last night. If you are using multiple detectors, then the same image is passed on each processing loop. This allows me to display a single image on the UI with the results per detector.

I've also added some more documentation on how I setup my Node-Red and Home Assistant notifications.

More notes on what's included in this release can be found here.

[jakowenko]

More updates 🚀

Frigate's config.yml file inspired me to do the same.

• I've migrated the ENV variables to a more readable config.yml file.
  Here's a small sample:

  server:
    port: 3000
  
  mqtt:
    host: 192.168.1.1
  
  home_assistant:
    url: http://192.168.1.1:8123
    token: xxx.xxx-xxx
  
  confidence:
    match: 60
    unknown: 40
  
  save:
    matches: true
    unknown: true
  
  frigate:
    url: http://192.168.1.1:4000
  
  detectors:
    compreface:
      url: http://192.168.1.1:8000
      key: xxx-xxx-xxx-xxx-xxx
    deepstack:
      url: http://192.168.1.1:8001

• Added Home Assistant support. Entities for people and cameras will automatically be created

Any feedback or feature suggestions would be appreciated.

[Alfiegerner]

I'm currently testing out compreface and deepstack with this, working really well thanks.

I find compreface more reliable and faster, and thinking there's little value running both. What's your experience with the different detectors @jakowenko ?

[jakowenko]

@Alfiegerner I also run CompreFace and DeepStack side by side. CompreFace is the clear winner for me too. I agree, if you're happy with the results from CompreFace, then there's not much value in running DeepStack.

I do want to add support for object recognition in the future which DeepStack supports. But that's still a little ways out. Need to figure out how to best fit it in with the UI.

[jakowenko]

Just released v0.6.0 which now includes the processing of Frigate's snapshot MQTT image.

Added

• UI: /config route to make live config changes
• UI: /files route to view currently trained images
• API: process Frigate snapshot MQTT messages
• API: support for CompreFace plugins

[osos]

What is the best way to install CompreFace and/or Deepstack to use with double-take?

Could you provide full docker-compose examples with double-take and compreface ?
Any specific settings you recommend?

[jakowenko]

Here's what a Double Take and DeepStack docker-compose file would look like.

version: '3.7'

services:
  double-take:
    container_name: double-take
    image: jakowenko/double-take:beta
    restart: unless-stopped
    volumes:
       - ${PWD}/double-take/storage:/.storage
    ports:
      - 3000:3000

  deepstack:
    container_name: deepstack
    image: deepquestai/deepstack
    restart: unless-stopped
    environment:
      VISION-FACE: 'True'
      VISION-DETECTION: 'True'
      MODE: High
    volumes:
      - ${PWD}/deepstack:/datastore
    ports:
      - 5000:5000

For CompreFace, I use the docker-compose and .env file from their GitHub

[D34DC3N73R]

If I have frigate set up in double-take, do I need to also define cameras in double-take? My current config is below for reference

# Double Take
server:
  port: 3000

mqtt:
  host: 192.168.0.178
  username: user
  password: pass
  topics:
    frigate: frigate/events
    matches: double-take/matches
    cameras: double-take/cameras

confidence:
  match: 60
  unknown: 40

objects:
  face:
    min_area_match: 10000

save:
  matches: true
  unknown: true

purge:
  matches: 168
  unknown: 8

frigate:
  url: http://192.168.0.178:5000 #doesn't work with http://frigate:5000 
  image:
    height: 500
  attempts:
    latest: 10
    snapshot: 0
  cameras:
    - front1
    - front2
    - back1
    - back2
    - side1
    - side2
    - doorbell
#  zones:
#    - camera: driveway
#      zone: zone-1

#cameras:
#  driveway:
#    snapshot:
#      topic: driveway/snapshot
#      url: http://192.168.1.1/latest.jpg

detectors:
#  compreface:
#    url: http://192.168.1.1:8000
#    key: xxx-xxx-xxx-xxx-xxx # key from recognition service in created app
  deepstack:
    url: http://deepstack:5000
#    key: xxx-xxx-xxx-xxx-xxx # optional api key
#  facebox:
#    url: http://192.168.1.1:8002

time:
  format: F
  timezone: America/Los_Angeles

sidenote, it seems like double-take has an issue connecting to frigate when using docker dns.
frigate:
url: http://frigate:5000

frigate:
url: http://192.168.0.178:5000

$ docker exec -it double-take ping frigate:5000
PING frigate:5000 (172.18.0.31): 56 data bytes
64 bytes from 172.18.0.31: seq=0 ttl=64 time=0.157 ms
64 bytes from 172.18.0.31: seq=1 ttl=64 time=0.064 ms
64 bytes from 172.18.0.31: seq=2 ttl=64 time=0.046 ms
64 bytes from 172.18.0.31: seq=3 ttl=64 time=0.052 ms
^C
--- frigate:5000 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.046/0.079/0.157 ms

[D34DC3N73R]

Out of curiosity, have you tried running compreface with GPU acceleration (0.5.0-arcface-r100-gpu)? When I run this with double-take, it's taking up over 2GB of GPU memory. Seems like a lot. I was just wondering if that's typical or if there's something else going on.

[jakowenko]

I haven't tried running the GPU version. I don't think it would work on my old MBP that is currently running CompreFace, but I'll look into it.

If you run the GPU version and stop Double Take and manually process images using the CompreFace API does the memory still increase? That will at least point you in the right direction of where it's coming from.

I saw you opened an issue on CompreFace's GitHub too, I'll follow along with that and see what the outcome is.

How's Double Take working for you?

// Edit: Saw the issue you opened is closed, seems like it's expected right now from the CompreFace GPU version.

[D34DC3N73R]

Yeah, I've switched to the mobile GPU version. It does seem much less accurate but that's expected I guess. I'm using a P2000 right now so I only have 5GB of GPU memory. I might switch back to the non-gpu version and compare accuracy.

Double-Take has been working great. I realize the limitations are that of the processors and not of double-take. I am getting lots of false positives at the moment, but I figure that's just par for the course until more photos are added. I'm wondering how many photos should be used to train. I'm also thinking it might be best to train using a single camera source.

[Eddie1974MAX]

Here's what my HA notification looks like. Replace the IP-ADDRESS below with yours. Let me know if this works for you. It will only send a notification when the state has changed for the trigger entities.

alias: Notify
description: ''
trigger:
  - platform: state
    entity_id: sensor.double_take_david
condition:
  - condition: template
    value_template: '{{ trigger.to_state.state != trigger.from_state.state }}'
action:
  - service: notify.mobile_app_iphone
    data:
      message: >-
        {{trigger.to_state.attributes.friendly_name}} is near the
        {{trigger.to_state.state}} @
        {{trigger.to_state.attributes.match.confidence}}% by
        {{trigger.to_state.attributes.match.detector}}:{{trigger.to_state.attributes.match.type}}
        taking {{trigger.to_state.attributes.attempts}} attempt(s) @
        {{trigger.to_state.attributes.duration}} sec
      data:
        attachment:
          url: >-
            http://IP-ADDRESS:3000/api/storage/matches/{{trigger.to_state.attributes.match.filename}}?box=true
mode: single

what's sensor.double_take_david?

[jakowenko]

Here's what my HA notification looks like. Replace the IP-ADDRESS below with yours. Let me know if this works for you. It will only send a notification when the state has changed for the trigger entities.

alias: Notify
description: ''
trigger:
  - platform: state
    entity_id: sensor.double_take_david
condition:
  - condition: template
    value_template: '{{ trigger.to_state.state != trigger.from_state.state }}'
action:
  - service: notify.mobile_app_iphone
    data:
      message: >-
        {{trigger.to_state.attributes.friendly_name}} is near the
        {{trigger.to_state.state}} @
        {{trigger.to_state.attributes.match.confidence}}% by
        {{trigger.to_state.attributes.match.detector}}:{{trigger.to_state.attributes.match.type}}
        taking {{trigger.to_state.attributes.attempts}} attempt(s) @
        {{trigger.to_state.attributes.duration}} sec
      data:
        attachment:
          url: >-
            http://IP-ADDRESS:3000/api/storage/matches/{{trigger.to_state.attributes.match.filename}}?box=true
mode: single

what's sensor.double_take_david?

That's the entity that gets created by Double Take once that user is found. It should follow this format: sensor.double_take_<name>

[jakowenko]

I published release v0.10.0 last night which includes the following.

Added

• Ability to enable authentication, which adds /login, /logout, and /tokens routes to the UI
• Ability to create access tokens so third party applications can authenticate with the API
• Ability to remove trained folder if no trainings exist
• Include token in published MQTT topics for accessing authenticated images
• Ability to adjust CompreFace det_prob_threshold value
• Web app standalone support with favicon

Changed

• Use normal web history for UI instead of hash history
• Increase wait time for API restarts
• Drop VUE_APP_API_URL env in favor of window.location.origin
• Revert to default publicPath when building UI

Fixed

• Replace spaces in subject names when publishing to MQTT
• Verify detectors are present on /config route when rendering status icons

[D34DC3N73R]

Do you have an example or documentation on how to use authentication? Also, an nginx reverse proxy example?

[jakowenko]

To enable authentication, just add auth: true to your config.yml file. You should then be prompted to create a password. Unless you're doing something fancy, everything should just work when you use auth. Tokens are passed to HA entities and MQTT topics to render the matched images. And if something falls outside of the scope or what Double Take does, then you can create access tokens via the UI to invoke the API manually.

I don't have a reverse proxy example handy, but can help if you run into any issues. I use Traefik as my reverse proxy.

[D34DC3N73R]

Thanks, I think I got it working but I'm waiting to make sure home assistant notifications will work with the FQDN. This is what I'm using with swag:

server {
    listen 443 ssl;
    listen [::]:443 ssl;

    server_name doubletake.*;

    include /config/nginx/ssl.conf;

    client_max_body_size 0;

    location / {
        include /config/nginx/proxy.conf;
        resolver 127.0.0.11 valid=30s;
        set $upstream_doubletake double-take;
        proxy_pass http://$upstream_doubletake:3000;
    }
    location /api {
        include /config/nginx/proxy.conf;
        resolver 127.0.0.11 valid=30s;
        set $upstream_doubletake double-take;
        proxy_pass http://$upstream_doubletake:3000;
   }
}

[jakowenko]

Awesome, glad you got it working! Do you need both entries if the UI and API are run from the same container? I thought just the / location would also satisfy the /api one. But you would know better.

Let me know if there's anything I can do to help.

[D34DC3N73R]

yes, I had tried with just the single / location, but saw some errors in chrome console. no errors since adding the /api location. There are a lot of similar configurations in the swag proxy-conf examples.

[DrSpaldo]

I've been trying out the Double-Take. Unfortunately it hasn't been very reliable for recognition; I have tried Deepstack, Facebox & CompreFace. CompreFace seems to win but still not great.

In relation to the snapshot size, does anyone know where Frigate generates the image from? Ie. Can I force it to use my HD stream to generate the snapshot? It appears it is using the secondary stream, so quite low res

[DrSpaldo]

Hi @ozett , thanks for the reply. I more so interested what setting does it take the snapshot size from...

[jakowenko]

Hey @DrSpaldo, thanks for checking out the project. The quality of the training photos plays a big factor in the reliability of the recognition. When I first started this project I used a lot of the camera images from Frigate to then train the detectors. I've since stopped doing this and only train using images from my phone. It's pretty easy to do from the UI and allows you to use the front facing camera.

Since doing this, the amount of false positives I've had has gone way down. I have my confidence match % set to 90 as well.

[DrSpaldo]

Hi @jakowenko , thanks for the reply and the work you have done so far. That is interesting, the way I was training was the wrong way for sure! Is there any easy way to wipe clean what I have already done to start the training again?

[jakowenko]

@DrSpaldo the train tab on the UI lets you untrain all the images for a person. If you're using just CompreFace, you could also create a new API key to make sure it's fresh.

[DrSpaldo]

@DrSpaldo the train tab on the UI lets you untrain all the images for a person. If you're using just CompreFace, you could also create a new API key to make sure it's fresh.

Thanks for the reply @jakowenko, I have done untrain on all my people and also generated the new API. It still shows people/names in the matches section but I hope that will change once I re-do the training process with real photos.

[ozett]

min is now getting snaps over http, so there is api-descprition for setting this with the size-query: https://blakeblackshear.github.io/frigate/usage/api#apieventsidsnapshotjpg

[DrSpaldo]

Hi @ozett, sorry. Maybe I am not being clear. I was wondering what stream the snapshots are being generated from? Ie. If I have my main stream for say record & detect, but, my sub-stream for RTMP. Are the snapshots generated from record, detect, rtmp, etc? Which subsequently means what resolution they will be

[blakeblackshear]

It's the detect stream, because it is the only stream decoded.

[DrSpaldo]

It's the detect stream, because it is the only stream decoded.

Thanks @blakeblackshear

[alexdelprete]

Frigate + Double Take is an amazing combo. IMHO, Double Take should be merged into Frigate, they fit perfectly together.

@blakeblackshear @jakowenko did you ever discuss the possibility to merge the two in a single project?

[jakowenko]

I'm flattered you are enjoying Double Take. I haven't spoken to Blake, but I'm guessing the DT feature set is out of scope for what he wants to support with Frigate. Unless there's plans to handle facial recognition in the future.

The code base is also very different, the DT backend is in Node.js and I believe the Frigate backend is in Python, so no easy way to merge unless someone rewrites it. Maybe someone will be inspired and want to build this into Frigate directly since there's a larger audience base. I'm a Vue/Node developer though, building Double Take has been a fun journey into object / facial detection. I'm glad others are finding it useful.

[jakowenko]

1.0.0 is officially out with support for web sockets, pagination, updated filtering, and more!

The configuration section of the README is also updated with better examples.

Thank you to everyone who has used and help test Double Take over the past few months and for all the awesome feature suggestions. Keep them coming!

[jakowenko]

v1.2.0 is hot off the shelf.

• Themes can now be set via dropdowns on the config page
• New log page allows viewing of container logs via the UI
• MQTT status icon

[blakeblackshear]

@jakowenko should we look at adding an endpoint for frigate that returns the image data in raw rgb and cropped? That avoids the CPU overhead and data loss of frigate converting to jpg only to have doubletake immediately decode it again and convert to raw rgb.

[jakowenko]

@jakowenko should we look at adding an endpoint for frigate that returns the image data in raw rgb and cropped? That avoids the CPU overhead and data loss of frigate converting to jpg only to have doubletake immediately decode it again and convert to raw rgb.

That would be awesome @blakeblackshear. Would you like me to open a feature request for this?

[blakeblackshear]

That would be great.

[cartersdad2008]

@jakowenko was there ever a feature request opened for this? I can see this helping processing time and overhead.

[NickM-27]

@cartersdad2008 See #1903

[tubalainen]

Both Double-Take and Frigate just got featured on Linus Tech Tips on YouTube: https://www.youtube.com/watch?v=B635wcdr6-w

[jakowenko]

v1.4.0

Bug Fixes

• api: validate mime type and result of database.get.fileByFilename
• api: catch facebox error before normalizing data
• api: delete orphaned db records when deleting training folder / files
• ui: fix multiselect dropdowns when open and scrolling
• ui: theme wouldn't load if setting to same theme

Features

• support for ARMv7 architecture
• api: api/latest/<name>.jpg
• api: use camera name when using custom mqtt topic
• ui: enable/disable sockets on matches page

[jakowenko]

I've recently published version 1.12.0 with includes support for AWS Rekognition and a few other things. I'm still really happy with CompreFace, but it's been cool to run it alongside a paid service to see how the two compare.

1.12.0 (2022-06-13)

Features

• api: opencv preprocess face check (ed30ad1)
• aws rekognition support (7904852)
• detectors: process images from specific cameras (5d39d0c)
• frigate: sort sub labels alphabetically #217 (82d8736)
• frigate: stop_on_match config option to break process loop (4b98990)
• opencv: adjust classifier settings via config (2e6c512)
• ui: show config errors (ddcaf89)
• ui: upload images to process with detectors (f774406)

Bug Fixes

• api: sleep if image hasn’t changed during retry loop (78e9808)
• better handling of update check (6de1cf1)

[darazi]

@jakowenko ,

Been using DoubleTale for the past week or so. Loving it, amazing work! Many thanks for putting this together.

As a feature idea, it would be awesome if the face-specific sensor (the one DoubleTake generates) would also send out a “detected” signal, Like a face specific motion detector (along with its current payload). It would make detection based automations possible. If I understand the sensor correctly, it currently returns a “last camera that detected this face” payload.

Out of curiosity. How did you find AWS's accuracy relative to CompreFace?

[yayitazale]

I have a qustion, can I use double take to recognize bird species with deepstack instead of human faces?

Thanks!

[jvrobert]

Quick question is doubletake going to be developed in the future? Not complaining, works great but if not maybe at some point frigate could subsume its functionality and integrate with compreface.

[DrSpaldo]

Quick question is doubletake going to be developed in the future? Not complaining, works great but if not maybe at some point frigate could subsume its functionality and integrate with compreface.

Looks like @jakowenko has gone AWOL. This project has been forked by @skrashevich and is being updated, try this instead:

https://github.com/skrashevich/double-take

[jakowenko]

Hey, I'm still around, life just got in the way for a bit. I'll have to catchup with @skrashevich and see if he'd like to collaborate more on it.

[skrashevich]

Hey, I'm still around, life just got in the way for a bit. I'll have to catchup with @skrashevich and see if he'd like to collaborate more on it.

We can continue works on this project together. Write to me in telegram (the username is the same as on the github)

[DrSpaldo]

Great news that @jakowenko is still around and that there may be some form of collaboration with @skrashevich

[jvrobert]

Great news, didn't know about @skrashevich fork, will give it a try.