Skip to content

Evaluation, Reproducibility, Benchmarks Meeting 17

AReinke edited this page Jun 22, 2022 · 1 revision

Minutes of meeting 17

Date: 22nd June 2022

Present: Carole, Michela, Jorge, Keyvan, Lena, Annika


TOP 1: General update

  • WG Members will distribute the Metrics Reloaded survey among their labs Michela sent emails regarding the challenge data integration but did not get any replies yet Jorge will contact Nicola regarding metrics for memory consumption and runtime and will handle this point Points to discuss in Metrics Reloaded: Cloud computing

TOP 2: Metric implementation

  • Annika and Carike prepared a table and check which metrics are already implemented in MONAI
  • Carole prepared issues and forwarded them to the MONAI dev team
    • For example, metrics are mostly implemented for the binary case; multi-threshold metrics or metrics at a specific threshold need to be reworked/implemented
  • Testing process: Carole will propose them to use the pitfall figures as test cases for validating the implementations
  • Carole will think about the idea to publish a MONAI Metrics Reloaded Package

TOP 3: Instantiation of biomedical use cases for Metrics Reloaded

  • The Nature Methods editor really likes the use cases but is not convinced that it make sense to cover biology and medical domains in one framework
  • We will update Fig 12 of the paper (https://arxiv.org/abs/2206.01653)
  • Figure should convey the message that problems are very similar and lead to same recommendations (or only slight variations that are highlighted)
    • Option: Add a new column with properties that are shared among examples (e.g. large organs)
    • Option: One medical, one biological example per task
  • We will tweet and ask the community which use cases we should covor
Clone this wiki locally