COSMOduleS: Classification, Object detection, Segmentation MOduleS

Github: source

Introduction

This repo provides comprehensive preprocessing and post-processing tools for common Computer Vision tasks.

Tasks	Subtasks	Defined Format	Visualization	Format Conversion	Output Analysis	Label Merging	Active Learning
Classification	binary1 binary-bg2 multi-class1 multiclass-bg2 multi-binary3	single_label1 single_label_bg2 multi_label3	-	-	metrics plotting export	ALL	Entropy
Detection	-	coco voc yolo GENERAL	ALL	between ANY two types	metrics plotting export	V	horizontal flip
Segmentation	instance1 semantic2	coco1+2 GENERAL1+2	ALL	coco2general	metrics plotting export	-	instance semantic

• "bg" means background. If there is background class, it must be class 0 in this repo.
• Adding prediction results after the defined format can use the visualization and output analysis. All the formats with predictions are in example/*/prediction, e.g. here.

Motivation

• [Classification] Complicated tasks

task	label idx min	compute class-0 metrics	threshold optimization	data format
binary classification	0	V	V	single_label
binary classification (cls-0 background)	1		V	single_label_background
multi-class classification	0	V		single_label
multi-class classification (cls-0 background)	1		V	single_label_background
multi-label classification (cls-0 background)	0		V	multi_label

• [Classification] threshold optimization

  • multi-class classification (cls-0 background) checks whether prob-cls-0 < threshold, if yes, the pd-cls is pd[1:].argmax()
  • multi-class classification (cls-0 background) and multi-label classification (cls-0 background) take the mean of all optimized threshold for each foreground class

• [Object Detection] Develop a GENERAL format to be the most convenient.

The formats can be summarized as following:

format	extension	files	type	box	disadvantage
coco	.json	1	int	(xmin, ymin, w, h)	get label of an image
yolo	.txt	len(imgs)	float	(cx, cy, w/2, h/2)	visualization, compute metrics, etc.
voc	.xml	len(imgs)	int	(xmin, ymin, xmax, ymax)	get class list
general	.json	1	int	(xmin, ymin, xmax, ymax)	NO

• [Segmentation] Develop a GENERAL format to be the most convenient.

Includes	Content	Advantage
general.json	Includes every imgs: path, contour, filled and boxes with class	Searching
gt_contour_*.npy	(H, W) with {0, 1, ..., num_classes} int	Plotting
gt_filled_*.npy	(num_classes, H, W) with 0 or 1 int values	Compute IOU for Metrics
*.jpg	Raw data	-

• Segmentation prediction format: (num_classes, H, W) with 0~1 float values (probability). e.g. here

Installation

pip install cosmodules

or

git clone https://github.com/bnbsking/COSMOduleS.git
pip install -e .

Quick Start - Classification

• Output Analysis:
  • Please conform your data format as either of one
    • single_label
    • multilabel
    • single_label_background
  • The analysis pipeline is at here
  • See more in the example

from cosmodules.classification import ClassificationAnalysis

ClassificationAnalysis(
    ant_path = "example/classification/data/single_label.json",
    save_folder = "example/classification/output/single_label",
)

• Label Merging:

from cosmodules.classification import ClassificationLabelMerging

ClassificationLabelMerging(
    cfg_path_list = [
        "example/classification/data/single_label.json",
        "example/classification/data_another_labeler/single_label.json",
    ],
    save_path = f"example/classification/output/label_merging/single_label.json"
)

• Active Learning (see more in the example):

from cosmodules.classification import ClassificationActiveLearning

ClassificationActiveLearning(
    pred_path = "example/classification/prediction/single_label.json",
    save_path = "example/classification/output/active_learning/single_label.json",
    loss_name = "entropy"
)

Quick Start - Object detection

• Format Conversion (see more in the example)

from cosmodules.detection import coco2any

coco2any(
    tgt_foramt = "voc",
    img_folder = "example/detection/data/coco",
    ant_path = "example/detection/data/coco/coco.json",
    save_folder = "example/detection/output/visualization_gt_conversion/coco2voc"
)

or

from cosmodules.detection import coco2general

coco2general(
    img_folder = "example/detection/data/coco",
    ant_path = "example/detection/data/coco/coco.json",
    save_path = "example/detection/output/visualization_gt_conversion/coco2general/general.json"
)

• Visualization (see more in the example)

from cosmodules.detection import show_coco

show_coco(
    img_name = "pic0.jpg",
    img_folder = "example/detection/data/coco",
    ant_path = "example/detection/data/coco/coco.json"
)

or

from cosmodules.detection import show_general

show_general(
    img_name = "pic0.jpg",
    ant_path = "example/detection/data/general.json",
)  # when the anntotation includes predictions it will be shown!

• Output Analysis
  • Please use the above Format conversion to change data format as general
  • The analysis pipeline is at here

from cosmodules.detection import DetectionAnalysis

DetectionAnalysis(
    ant_path = "example/detection/data/general.json",
    save_folder = "example/detection/output/metrics"
)

• Label Merging:

from cosmodules.detection import DetectionLabelMerging

DetectionLabelMerging(
    cfg_path_list = [
        "example/detection/data/general.json",
        "example/detection/data_another_labeler/general.json",
    ],
    save_path = "example/detection/output/label_merging/general.json",
    ties_handling = "union"
)

• Active Learning:

from cosmodules.detection import DetectionActiveLearningByHFlip

DetectionActiveLearningByHFlip(
    pred_path_1 = f"{ROOT}/example/detection/prediction/general.json",
    pred_path_2 = f"{ROOT}/example/detection/prediction/general_horizontal_flip.json",
    save_path = f"{ROOT}/example/detection/output/active_learning/general.json"
)

Quick Start - Segmentation

• Format Conversion (see more in the example)

from cosmodules.segmentation import coco2general

coco2general(
    img_folder = "example/segmentation/data/coco",
    ant_path = "example/segmentation/data/coco/coco.json",
    save_folder = f"example/segmentation/output/visualization_gt_conversion/coco2general"
)

• Visualization (see more in the example)

from cosmodules.segmentation import show_coco

show_coco(
    img_name = "img1.jpg",
    img_folder = "example/segmentation/data/coco",
    ant_path = "example/segmentation/data/coco/coco.json"
)   # when the anntotation includes predictions it will be shown!

or

from cosmodules.segmentation import show_general

show_general(
    img_name = "img1.jpg",
    ant_path = "example/segmentation/data/general/general.json"
)

• Output Analysis
  • Please use the above Format conversion to change data format as general
  • The analysis pipeline is at here

from cosmodules.segmentation import SegmentationAnalysis

SegmentationAnalysis(
    ant_path = "example/segmentation/prediction/instance/general.json",
    save_folder = "example/segmentation/output/metrics/instance",
    task = "instance",
)

or

from cosmodules.segmentation import SegmentationAnalysis

SegmentationAnalysis(
    ant_path = "example/segmentation/prediction/semantic/general.json",
    save_folder = "example/segmentation/output/metrics/semantic",
    task = "semantic"
)

• Active Learning:

from cosmodules.segmentation import (
    InstanceSegmentationActiveLearningByHFlip,
    SemanticSegmentationActiveLearning
)

InstanceSegmentationActiveLearningByHFlip(
    pred_path_1 = "example/segmentation/prediction/instance/general.json",
    pred_path_2 = "example/segmentation/prediction/instance_horizontal_flip/general.json",
    save_path = "example/segmentation/output/active_learning/instance.json"
)

or

SemanticSegmentationActiveLearning(
    pred_path = "example/segmentation/prediction/semantic/general.json",
    save_path = "example/segmentation/output/active_learning/semantic.json",
    loss_name = "entropy"
)

Examples

• [detection]: format conversion workflow

• detection visualization

• confusion

• prf curves

More

• Feel free to ask if you have any question.
• Notice not supported
  • segmentation general2coco
  • segmentation label merging

Acknowledgement

• Confusion Matrix reference here