[weighted_box_fussion] - an alternative for box_non_max_suppression

[hardikdava]

Search before asking

• I have searched the Supervision issues and found no similar feature requests.

Description

Current object detection models removes overlapping boxes by nms which can reduce accuracy of the final results. It can be avoided by Weighted Box Fusion which can accepts all the prediction whether from a single model or multiple models.

Reference: Weighted Box Fusion
Original Implementation: ensemble-boxes

Use case

import supervision as sv

wbf = sv.WeightedBoxFusion()

res_a = model_a(image)
det_a = sv.Detection(res_a)

res_b = model_b(image)
det_b = sv.Detection(res_b)

wbf_detections = wbf([det_a, det_b])

Additional

class WeightedBoxFusion:

	def __init__(self):
		pass
	
	def __call__(self, detections: Union[Detections, List[Detections]]) -> Detections:
		pass
		result_detections = Detections(...)
		return result_detections 

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[SkalskiP]

@hardikdava I have yet to hear of this post-processing approach. Looks interesting. IS it better than NMS?

[hardikdava]

@SkalskiP yes, it is better in terms of accuracy but not in terms of speed. You can learn about it more and direct comparision on this blogpost: https://learnopencv.com/weighted-boxes-fusion/

[AlainPilon]

@hardikdava Did you implement the WBF? Even if Supervision doesnt accept the feature, I could use it for my current project.

[SkalskiP]

@hardikdava, if you have that implemented somewhere already, I'd love to see it. I know that you are probably super busy, but some of the open-source contributors may be able to add it to our codebase.

[AlainPilon]

I found this: https://github.com/ZFTurbo/Weighted-Boxes-Fusion which does most of the work, we just need the detections coordinates to be as % instead of absolute pixel values. The packages also does NMS which would be a feature overlap with the existing Supervision code. I dont know how we should handle this:

• extract just want we need for WBF, or
• replace the existing NMS code with the one in the package (and get a few extra features for free).

My working version of Supervision has drifted quite a bit from the current release. Anything I do on my side would probably not be mergeable in main.

[hardikdava]

@AlainPilon Yes, I took reference from that project. @SkalskiP I can not contribute to open source projects due to my company policy. but I haven't started working on it. It was just an idea. But from my knowledge, it does not make effect on accuracy. The detection accuracy can boost by using SAHI.

[SkalskiP]

I can not contribute to open-source projects due to my company policy.

Sad to hear it :/ We miss you!

[LinasKo]

We're reviving this one.

Some context for new contributors:

SkalskiP:
The InferenceSlicer consists of two key elements: a moving window that slices the image into smaller, partially overlapping patches and a postprocessing algorithm that primarily addresses duplicate detections occurring at the edges of patches. You can see the InferenceSlicer in action in this how-to guide.

Currently, we use non-max suppression to filter out double detections, but there are other, often better, methods for handling these extra detections. NMS removes overlapping detections, but there are methods that involve merging them instead.

We are now looking at two methods to solve this:

• This PR (weighted box fusion)
• Add Non-Maximum Merging (NMM) to Detections #500

We don't know which is better, how it compares to our current with_nms solution, or what tradeoffs there would be. Perhaps we'd include both in the InferenceSlicer, perhaps only one.

We'd like to test it with images, videos, also an example where no-detection is being merged with some-detections.

For testing with different models, some inspiration can be found here: https://supervision.roboflow.com/develop/how_to/detect_small_objects/

Code-wise, the implementation would result in a Detections.with_weighted_box_fusion function inside supervision/detection/core.py, similar to Detections.with_nms we have now, and some files to test the results. If there's more complexity, we can add code in the detection/utils.py.

[LinasKo]

@sharingan000, how does this sound? Would you have the time to help out?
I'm here to help if needed 😉