Semantic genai tagging of recordings even when detection is disabled and motion is enabled

[saket424]

@hunterjm and @NickM-27
I just tried the semantic genai tagging enhancement using the 0.14.1 branch . Great start!

Even when frigate is used purely to record a busy street scene where object detection is disabled since it has a tendency to spike the CPU to uncontrollable levels, it makes sense to summarize a recording by creating a manual event and adding the applicable metadata. So it is searchable on a timeline

Thoughts?

[NickM-27]

Running on recordings is a lot more complicated because you need an image to send. Also, needs something to be searchable. Could be manual event but that presents additional challenges and I'm not sure what the trigger would be. Perhaps the manual event API could have a use genai param but not sure if it would work well.

[saket424]

Once the motion recording has ended (been created). we can sample a still image every gap seconds using ffmpeg like amblegpt does and summarize the scene. There are other models like Florence 2 that do a great job of drawing bounding boxes on various detected objects within the snapshot scene to make the metadata quite rich. As a start, we can pick a point where there is motion, possibly the midpoint of the recording as the reference snapshot

Just a suggestion. This can of course be refined

[NickM-27]

My point is that that requires a lot more work, not a simple addition but basically an entire new feature and pipeline. I wouldn't expect something like that until we learn more about how people use this new search feature in frigate. Not using object detection in frigate is still atypical, and especially if you're going to start decoding and analyzing streams anyway might as well just enable object detection as it is in frigate already.

[saket424]

Just a sample snippet from amblegpt

def extract_frames_ffmpeg(video_path, gap_secs):
    logging.info("Extracting frames from video using FFmpeg")

    # Extract the video file name and create a directory for frames
    video_name = pathlib.Path(video_path).stem
    frames_dir = os.path.join("cache_video", video_name)
    os.makedirs(frames_dir, exist_ok=True)

    # FFmpeg command to extract frames every gap_secs seconds and resize them
    ffmpeg_command = [
        "ffmpeg",
        "-i",
        video_path,
        "-vf",
        f"fps=1/{gap_secs},scale=480:480:force_original_aspect_ratio=decrease",
        "-q:v",
        "2",  # Quality level for JPEG
        os.path.join(frames_dir, "frame_%04d.jpg"),
    ]

    # Execute FFmpeg command
    subprocess.run(ffmpeg_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

    # Read and encode the extracted frames
    frames = []
    for frame_file in sorted(os.listdir(frames_dir)):
        if not frame_file.endswith(".jpg"):  # to exclude .DS_Store etc
            continue
        with open(os.path.join(frames_dir, frame_file), "rb") as file:
            frame_bytes = file.read()
            frames.append(base64.b64encode(frame_bytes).decode("utf-8"))

    logging.info(f"Got {len(frames)} frames from video")
    return frames

def prompt_gpt4_with_video_frames(prompt, base64_frames, low_detail=True):
    logging.info("prompting GPT")
    headers = {
        "Content-Type": "application/json",
        "Authorization": f"Bearer {os.getenv('OPENAI_API_KEY')}",
    }
    PROMPT_MESSAGES = [
        {
            "role": "user",
            "content": [
                prompt,
                *map(
                    lambda frame: {
                        "type": "image_url",
                        "image_url": {
                            "url": f"data:image/jpeg;base64,{frame}",
                            "detail": "low" if low_detail else "high",
                        },
                    },
                    base64_frames,
                ),
            ],
        },
    ]
    payload = {
        "model": "gpt-4o-mini",
        "messages": PROMPT_MESSAGES,
        "max_tokens": 200,
    }

    return requests.post(
        "https://api.openai.com/v1/chat/completions", headers=headers, json=payload
    )

[saket424]

Just farm out the detection/summarization to something like openedai-vision or similar

https://www.reddit.com/r/MLQuestions/comments/1b9yh71/using_moondream_gpt4_to_classify_any_length_video/

[NickM-27]

That would require even more work, configuration, code to maintain, etc just to duplicate what already exists in frigate. I think at this point the suggestion is clear, it will take time to think through what approach we think makes most sense, and gather how such a feature would be most useful to users of frigate.

[NickM-27]

Also worth mentioning that all of the work frigate does currently is available to APIs, so what you're suggesting could be built as an external sidecar service that doesn't require changes in frigate itself.

[saket424]

I agree all this is possible in an external side car like nodered. I like frigate for it's clip management and centralized store , camera management and API. Let's see how people end up using semantic tagging and search and how things unfurl