Metadata filter value contains IN meta list []

[touhi99]

Hi everyone, I see there's a nice conditioning already existing in metadata filtering like here.

pipeline.run(
data={"retriever": {
"query": "Why did the revenue increase?",
"filters": { "operator": "AND",
"conditions": [
{"field": "meta.years", "operator": "==", "value": "2019"},
{"field": "meta.companies", "operator": "in", "value": ["BMW"]},
]
}
}
}
)

But, how do I retrieve if meta.companies field containg multiple values and i want to search among them which rows containing BMW?

for example, meta.companies in my data is a list
[bmw]
[mercedes, audi]
[bmw, volkswagen]

how to make sure, it retrieves 1st and 3rd row?

[vblagoje]

I'm not sure tbh. Looking through qdrant docs I would have guessed {"field": "meta[].companies", "operator": "==", "value": "BMW"} but let me check with my colleague @silvanocerza who worked on this feature

[touhi99]

Hi, was able to solve it by customizing the pgvector filter, adding a new filter logic and @> to search in contains

def _contains(field: str, value: Any) -> Tuple[str, Any]:
    if not isinstance(value, list):
        msg = f"{field}'s value must be a list when using '@>' comparator in Pinecone"
        raise FilterError(msg)

    json_value = json.dumps(value)

    return f"({field})::jsonb @> %s::jsonb", json_value

[vblagoje]

And using this approach you are able to search through array field in meta? Can you share the filter now?

[touhi99]

key in list_value_keys: condition = {"field": f"meta.{key}", "operator": "@>", "value": [value]}

I am running check on multiple fields here.

the idea was something like this to check:

SELECT *
FROM table
WHERE meta->>'Valid' = $1
AND (meta->'clients')::jsonb @> '["bmw"]';

[vblagoje]

Glad it works but imho this is a bit umfff, there has to be another way 🤦

[touhi99]

definitely could have, i just needed something urgent and hence i also posted if anyone knows :D so if anyone has some better solution would be great as well

[touhi99]

may also work with ANY i guess, but i had some error and didnt go forward

[bodavijay24]

Is this functionality been implemented for InMemoryDocumentStore() ?