What is the best way to extract recall and precision for a specific Intersection-over-Union threshold?

[bw4sz]

Hi all, thanks for this amazing work.

I mantain a python package for airborne object detection. I'd like to migrate from our custom IoU evaluation to this library. I see a the mAP and IoU metrics in the detection module.

Our current metric is the proportion of detections that overlap with targets of IoU > 0.4

Within a pytorch-lightning module I can define a metric

        # Metrics
        self.iou_metric = IntersectionOverUnion(
            class_metrics=True,
            iou_threshold=self.config["validation"]["iou_threshold"])
        self.mAP_metric = MeanAveragePrecision()

and call in during validation step

    def validation_step(self, batch, batch_idx):
        """Evaluate a batch
        """
        try:
            path, images, targets = batch
        except:
            print("Empty batch encountered, skipping")
            return None

        #Get loss from "train" mode from torchvision.retinanet, but don't allow optimization
        self.model.train()
        with torch.no_grad():
            loss_dict = self.model.forward(images, targets)
            
        # sum of regression and classification loss
        losses = sum([loss for loss in loss_dict.values()])

        self.model.eval()
        preds = self.model.forward(images)

        # Calculate intersection-over-union
        self.iou_metric.update(preds, targets)
        self.mAP_metric.update(preds,targets)

        # Log loss
        for key, value in loss_dict.items():
            self.log("val_{}".format(key), value, on_epoch=True)

        return losses
    
    def on_validation_epoch_end(self):
        output = self.iou_metric.compute()
        self.log_dict(output)
        self.iou_metric.reset()

        output = self.mAP_metric.compute()
        self.log_dict(output)
        self.mAP_metric.reset()

The docs are not clear to me, and I need a push in the right direction.

1. How are IoU metrics being aggregated on .compute()? Average? Is there a way to prevent aggregation and report the per-detection IoU score? Similiar to the average="none" reduce method in torchmetrics.accuracy.

This discussion (#520) hints there is an argument, but it does not seem exist anymore. There is no arg reduction in torchmetrics.detection.IntersectionOverUnion.

>>> from torchmetrics.detection import IntersectionOverUnion
>>> metric  = IntersectionOverUnion(reduction="none")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/benweinstein/.conda/envs/DeepForest/lib/python3.9/site-packages/torchmetrics/detection/iou.py", line 151, in __init__
    super().__init__(**kwargs)
  File "/Users/benweinstein/.conda/envs/DeepForest/lib/python3.9/site-packages/torchmetrics/metric.py", line 146, in __init__
    raise ValueError(f"Unexpected keyword arguments: {', '.join(kwargs_)}")
ValueError: Unexpected keyword arguments: `reduction`

This would allow me to calculate the recall and precision for the batch of detections.

2. Alternatively is there a way to extract the recall and precision along the curve used to calculate mAP?

self.mAP_metric.compute()
{'map': tensor(0.0422), 'map_50': tensor(0.1114), 'map_75': tensor(0.0254), 'map_small': tensor(0.), 'map_medium': tensor(0.0510), 'map_large': tensor(-1.), 'mar_1': tensor(0.0089), 'mar_10': tensor(0.0342), 'mar_100': tensor(0.0848), 'mar_small': tensor(0.), 'mar_medium': tensor(0.0944), 'mar_large': tensor(-1.), 'map_per_class': tensor(-1.), 'mar_100_per_class': tensor(-1.), ...}

The data is inside there to create the precision recall curve.

[tcotte]

Hello, I am not sure to understand well your first question but I can answer to the second one.
It exists a boolean attribute extended_summary (default=False) of the class MeanAveragePrecision which enables to return precision and recall when compute() is call.

metric = MeanAveragePrecision(iou_type="bbox", extended_summary=True)`

validation_metrics = metric.compute()

precision = validation_metrics['precision']

recall= validation_metrics['recall']

As you can read in the documentation:

• precision: a tensor of shape (TxRxKxAxM) containing the precision values. Here T is the
  number of IoU thresholds, R is the number of recall thresholds, K is the number of classes,
  A is the number of areas and M is the number of max detections per image.
• recall: a tensor of shape (TxKxAxM) containing the recall values. Here T is the number of
  IoU thresholds, K is the number of classes, A is the number of areas and M is the number
  of max detections per image.