Skip to content

Issues: embeddings-benchmark/mteb

Finalizing MMTEB
#784 opened May 22, 2024 by KennethEnevoldsen
Open 6
Hacktoberfest 🎉
#1279 opened Oct 6, 2024 by isaac-chung
Open
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

75 Open 345 Closed
75 Open 345 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Cannot reproduce exact numbers (MTEB French retrieval) from the leaderboard
#1314 opened Oct 23, 2024 by PxYu
2
FSTimeoutError for MLSUMClusteringP2P & MLSUMClusteringS2S
#1311 opened Oct 23, 2024 by bourdoiscatie
2
Scrollable dropdown
#1304 opened Oct 21, 2024 by x-tabdeveloping
Leaderboard - Overview Issue
#1303 opened Oct 21, 2024 by x-tabdeveloping
4 tasks
Got unsupported ScalarType BFloat16 in RerankingEvaluator.py bug Something isn't working
#1297 opened Oct 18, 2024 by xujunrt
2
Serialization and validation are slow
#1296 opened Oct 17, 2024 by x-tabdeveloping
5
Add older MMTEB baselines
#1295 opened Oct 17, 2024 by orionw
Remove encode_corpus and encode_queries and implement a "document" class
#1284 opened Oct 6, 2024 by KennethEnevoldsen v2.0.0
Transitioning retrieval datasets to retrieval format recommend in #1090 good first issue Good for newcomers hacktoberfest
#1282 opened Oct 6, 2024 by KennethEnevoldsen
Hacktoberfest 🎉 hacktoberfest
#1279 opened Oct 6, 2024 by isaac-chung
New benchmark interface
#1272 opened Oct 3, 2024 by KennethEnevoldsen v2.0.0
1
Add FinMTEB good first issue Good for newcomers hacktoberfest
#1267 opened Oct 1, 2024 by Muennighoff
3
Add documentation for custom rerankers hacktoberfest
#1266 opened Sep 30, 2024 by Muennighoff
3
Add PIRB good first issue Good for newcomers hacktoberfest
#1265 opened Sep 30, 2024 by Muennighoff
2
Only skip benchmarking if split results are the same too
#1260 opened Sep 29, 2024 by Muennighoff
1
Massive Multimodal Extension of MTEB
#1249 opened Sep 27, 2024 by Muennighoff 100+
1
SWE-Bench Retrieval
#1238 opened Sep 24, 2024 by Muennighoff
@john-b-yang
Add proper metadata to models
#1234 opened Sep 24, 2024 by x-tabdeveloping
@x-tabdeveloping @KennethEnevoldsen
5
Allow aggregated tasks within benchmarks
#1231 opened Sep 23, 2024 by KennethEnevoldsen
4
Unable to reproduce MTEB leaderboard results on TRECCOVID using Snowflake/snowflake-arctic-embed-m-v1.5
#1227 opened Sep 20, 2024 by AndreaSottana
2
Cross-Encoder Support for Reranking Abstract Task enhancement New feature or request help wanted Extra attention is needed
#1214 opened Sep 11, 2024 by orionw
Figure out an approach for adding experiments to the leaderboard
#1211 opened Sep 9, 2024 by KennethEnevoldsen
4
Re-add CQADupstack merge script / find other way to make merging easier
#1171 opened Aug 20, 2024 by Muennighoff
Add CRAG good first issue Good for newcomers new-dataset
#1151 opened Aug 11, 2024 by Muennighoff
@john-b-yang
[New dataset request] Please add MKQA good first issue Good for newcomers new-dataset
#1149 opened Aug 11, 2024 by PrithivirajDamodaran
8
ProTip! What’s not been updated in a month: updated:<2024-09-23.