-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Generating embeddings with ONNX Runtime leads to errors #2983
Comments
Hello! The former is an issue with Optimum it seems. I've reported it here: huggingface/optimum#2062 The second one I'm not familiar with. I also can't reproduce that one, but it seems that you're using the from sentence_transformers import SentenceTransformer
model = SentenceTransformer("sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2", backend="onnx", model_kwargs={"provider": "CPUExecutionProvider"})
sentences = ["This is an example sentence", "Each sentence is converted"]
embeddings = model.encode(sentences)
|
I'm adding testing of the top 10 SBERT multi-lingual models from MTEB, several issues came out: # top 10 MTEB leaderboard multilingual SBERT embeddings models
models = [
'BAAI/bge-multilingual-gemma2',
'intfloat/multilingual-e5-large-instruct',
'HIT-TMG/KaLM-embedding-multilingual-mini-v1',
'gte-multilingual-base',
'Alibaba-NLP/gte-multilingual-base',
'intfloat/multilingual-e5-base',
'intfloat/multilingual-e5-small'
]
for model_name in models:
try:
model = SentenceTransformer (model_name, backend="onnx",
model_kwargs={
"provider": "CPUExecutionProvider",
# not supported with onnx
#"torch_dtype": torch.float16
},
trust_remote_code=True,
cache_folder='/mnt/datasets/sbert')
'''
SentenceTransformer(
(0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: ORTModelForFeatureExtraction
(1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
(2): Normalize()
'''
print(model)
sentences = ["This is an example sentence", "Each sentence is converted" ]
embeddings = model.encode (sentences)
print( embeddings.shape() )
except Exception as e:
print(f'error loading {model_name} {str(e)}') Stacktrace
error loading intfloat/multilingual-e5-small 'NoneType' object has no attribute 'numpy'
|
With the new release of version 3.2.0, the use of ONNX has become much easier but initial local tests led to various errors, meaning that it was not possible to use ONNX Runtime via Sentence Transformers. See these two examples:
intfloat/multilingual-e5-small
Lead to following error:
sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
Lead to following error:
Local environment:
python=3.10
sentence-transformers=3.2.0
onnx=1.17.0
onnxruntime=1.19.2
optimum=1.23.0
The text was updated successfully, but these errors were encountered: