FAISSDocumentStore scores are always in same range

[mlnsharma]

I am using FAISSDocumentStore for a QA system. When the FAISSDocumentStore.query_by_embedding() is called (internally during pipeline retrieval) with top_k=10, the documents matched in the index have normalised scores is in the range 0.50 - 0.51, actual score in the range 0.72-0.76.

When a query passed doesn't have matching documents, the top 10 documents returned have scores in the same range as mentioned above. This is preventing me from determining if the query is indeed matching the document or not.

I am using the default settings for similarity, index factory, scale_score. Constructor call looks like below.

document_store = FAISSDocumentStore(index_path=<>, config_path=<>, duplicate_documents='skip', validate_index_sync=False)

How to determine if the score represents a lower/higher similarity match? Are there any other configs to get the score representing the actual similarity?

[ZanSara]

Hey @mlnsharma, I believe that also depends on the embeddings. How do you generate them? Are you sure the model actually generates good embeddings for your documents?

[mlnsharma]

Hi @ZanSara
I am using OpenAI + 'text-embedding-ada-002' model for embedding generation. Can you please suggest any specific settings or alternatives to get correct scores?

[anakin87]

I would suggest trying to set scale_score=False.

Related API reference