genai seems to work best with openai provider and not so well with the ollama provider

[saket424]

Describe the problem you are having

Only the openai provider implementation invokes the chat completion api whereas the ollama provider implementation does not

Is it possible to broaden the implementation of openai to override the https://api.openai.com/v1 to provide an alternate baseurl so any openai api compatible backend also works

Version

0.15.0

Frigate config file

allow for an alternate baseurl so projects like https://github.com/matatonic/openedai-vision can be in play as the backend

Relevant log output

the current implementation is hardwired to use https://api.openai.com/v1

Frigate stats

No response

Operating system

Debian

Install method

Docker Compose

Object Detector

Coral

Any other information that may be helpful

No response

[hawkeye217]

Thanks for your request. You're using an unreleased, uncompleted, and completely unsupported feature at this point, subject to much change. We're happy to review PRs at this point in time if you'd like to make a contribution.

[hunterjm]

It should be a pretty simple change since we already have base_url as a part of the config. You can override it already with the OPENAI_BASE_URL environment variable though.

[saket424]

yes @hunterjm your suggestion works as does

diff --git a/frigate/genai/openai.py b/frigate/genai/openai.py
index d0178df8..b0c1c6f0 100644
--- a/frigate/genai/openai.py
+++ b/frigate/genai/openai.py
@@ -18,7 +18,7 @@ class OpenAIClient(GenAIClient):
 
     def _init_provider(self):
         """Initialize the client."""
-        return OpenAI(api_key=self.genai_config.api_key)
+        return OpenAI(base_url=self.genai_config.base_url,api_key=self.genai_config.api_key)
 
     def _send(self, prompt: str, images: list[bytes]) -> Optional[str]:
         """Submit a request to OpenAI."""

[saket424]

the ollama provider does not seem to work. It calls api/generate which returns a vector but no text description. I was trying moondream:latest as the model ( https://ollama.com/library/moondream )

[hunterjm]

That must be the model then? Ollama works for me with the llava1.6 model.

[DrSpaldo]

That must be the model then? Ollama works for me with the llava1.6 model.

@hunterjm do you have any basic guide of how you installed Ollama? I've installed it a couple of times using different methods, none of them work with Frigate. Obviously I must be doing something wrong but was hoping you may have a method I can copy?

[saket424]

@hunterjm
I dont know what I am doing but the following change got things working using matatonic/openedai-vision and the moondream model. Without this change I was getting a 422 error. It appears api.openai.com is more tolerant and does not throw a 422 error even though the prompt does not have the required text type

diff --git a/frigate/genai/openai.py b/frigate/genai/openai.py
index d0178df8..d62143f2 100644
--- a/frigate/genai/openai.py
+++ b/frigate/genai/openai.py
@@ -18,7 +18,7 @@ class OpenAIClient(GenAIClient):
 
     def _init_provider(self):
         """Initialize the client."""
-        return OpenAI(api_key=self.genai_config.api_key)
+        return OpenAI(base_url=self.genai_config.base_url, api_key=self.genai_config.api_key)
 
     def _send(self, prompt: str, images: list[bytes]) -> Optional[str]:
         """Submit a request to OpenAI."""
@@ -39,7 +39,7 @@ class OpenAIClient(GenAIClient):
                             }
                             for image in encoded_images
                         ]
-                        + [prompt],
+                        + [{"type": "text", "text":prompt }],
                     },
                 ],
                 timeout=self.timeout,

[saket424]

@hunterjm
It is quite confusing that the semantic text search defaults to the mixed search domain where the either "description" or "thumbnail" can win in similarity and there is no way to specify the search domain (via API)

I took a crack at separating them by adding a search_source argument. Thoughts?

diff --git a/frigate/api/event.py b/frigate/api/event.py
index 525fb951..77a0cbf5 100644
--- a/frigate/api/event.py
+++ b/frigate/api/event.py
@@ -275,6 +275,7 @@ def event_ids():
 def events_search():
     query = request.args.get("query", type=str)
     search_type = request.args.get("search_type", "text", type=str)
+    search_source = request.args.get("search_source", "description", type=str)
     include_thumbnails = request.args.get("include_thumbnails", default=1, type=int)
     limit = request.args.get("limit", 50, type=int)
 
@@ -407,17 +408,27 @@ def events_search():
 
     results = {}
     for event_id in thumb_ids.keys() | desc_ids:
-        min_distance = min(
-            i
-            for i in (thumb_ids.get(event_id), desc_ids.get(event_id))
-            if i is not None
-        )
-        results[event_id] = {
-            "distance": min_distance,
-            "source": "thumbnail"
-            if min_distance == thumb_ids.get(event_id)
-            else "description",
-        }
+        if search_source == "thumbnail":
+            distance = thumb_ids.get(event_id)
+            source = "thumbnail"
+        elif search_source == "description":
+            distance = desc_ids.get(event_id)
+            source = "description"
+        elif search_source == "both":
+            thumb_distance = thumb_ids.get(event_id)
+            desc_distance = desc_ids.get(event_id)
+            if thumb_distance is not None and (desc_distance is None or thumb_distance <= desc_distance):
+                distance = thumb_distance
+                source = "thumbnail"
+            else:
+                distance = desc_distance
+                source = "description"
+
+        if distance is not None:
+            results[event_id] = {
+                "distance": distance,
+                "source": source
+            }
 
     if not results:
         return jsonify([])

[hunterjm]

I had something similar in the 0.13 version of this implementation, but didn’t port it over. I think it’s a good idea to add as a filter. I would actually just update the search_type parameter to take description and both as options. I would also implement these checks further up so we aren’t actually doing the expensive operation of querying the embeddings that we already know we won’t return.

feel free to implement and open as a PR and I’ll happily review.

[saket424]

@hunterjm
You have done some great work and quite a bit of this is over my head and so I would find it hard to send you a coherent PR.
It would be wonderful if you can port forward the changes you did in 0.13. Sometime when we do not know what we are looking for apriori or the description is absent or does not do justice due to an inadequate prompt, thumbnail embeddings are all we have and so having the ability to search via thumbnail/description/both could be very useful if exposed in the UI.

I endorse your previous direction that with a good CLIP model we can get quite far
#8998 (comment)

add the ability to choose searching text by thumbnail or description

There are some new CLIP models that are probably superior to what we had previously such as this one from Google
CLIP_PATH = "google/siglip-so400m-patch14-384"
used here https://huggingface.co/spaces/fancyfeast/joy-caption-pre-alpha/blob/main/app.py
The ability to try other CLIP models will also be very nice to have besides the CLIP ViT-B/32 model in use now