Script for evaluation of GermanDPR

[gabiBettgenhaeuser]

Hi deepset folks, I love your contributions for the German NLP community.
In Deutsche Telekom, we have open sourced recently a German QA model (https://huggingface.co/deutsche-telekom/bert-multi-english-german-squad2).
Currently, we are also training a german DPR model with an internal high quality dpr dataset which contains over 90k question/answers pairs - hopefully, we can open source this model too.
We would like to run the same evaluation that you did in: https://huggingface.co/deepset/gbert-base-germandpr-reranking.
Did you share the evaluation code for it somewhere?

[Timoeller]

Hey really cool work.

QA Data and model

Will you also open source the translated data next to the deutsche-telekom/bert-multi-english-german-squad2 model?
Btw we based our https://huggingface.co/deepset/gelectra-base-germanquad on an automatically translated SQuAD, but without human correction. Have you evaluated your model on the GermanQuAD test set?

DPR evaluation

I dont fully understand what you mean. The model you linked is for reranking, DPR is mostly used at indexing time - you can of course rerank with DPR, Elasticsearch or any other retriever as well but it does not make so much sense since you can apply retrieval on the whole corpus directly. I assume you want to evaluate your DPR model and compare to existing german retrieval models?

We have open sourced https://huggingface.co/deepset/gbert-base-germandpr-ctx_encoder and https://huggingface.co/deepset/gbert-base-germandpr-question_encoder based on our German QA and DPR dataset. For the evaluation we used Haystacks Evaluation, based off our tutorial. When you use the tutorial you should index the whole wikipedia corpus and not just the test set documents.
For evaluation on the whole wikipedia I can recommend https://github.com/UKPLab/beir#beers-multilingual-datasets. Nandan (the main author of BeIR) and we will also host an NLP meetup soon where we described the methods used. We would be delighted to see you there and exchange experiences.