Evaluation, Reproducibility, Benchmarks Meeting 7

Minutes of meeting 7

Date: 27th January 2021

Present: Carole, David, Annika, Jens, Paul, Nicola, Keyvan, Bennett, Michela

---

TOP 1: Task forces reports

Easy data access task force (Michela)

• Survey and email ready to be sent
• Will be sent to Lena

Metrics task force (Carole):

• Currently on hold
• Many tickets are currently taken care of, especially regarding the reporting

Benchmarking task force (Annika):

• Currently on hold

---

TOP 2: MICCAI challenges

• Final feedback of acceptance for MICCAI challenges: Beginning of march
• Suggestion: Challenge organizers may implement baselines in MONAI
  • Help participants with dataloading, structuring and understanding the challenge
  • Decision: Contact challenge organizers to inform them about this possibility and clarify the possible benefit

---

TOP 3: Delphi process on metrics

• Round 1
  • 25 experts replied to questionnaire
  • 5 organizers will take care of evaluation
  • Extracted multiple scenarios and will assign categories (e.g. Object detection, classification etc.)
  • Be aware of biases in the replies (different research fields etc.)
• Round 2
  • Keep structure of first questionnaire (only few text fields; easy to fill out)
  • Goal: Assign metric short list to categories/applications and collect pitfalls
• Rounds 3+
  • Which metrics should (not) be combined?
  • Metric with synonyms: Decide which wording to use
  • Clinical relevance of individual metrics