Boundary IoU: Improving Object-Centric Image Segmentation Evaluation

By Bowen Cheng, Ross B. Girshick, Piotr Doll'ar, Alexander C. Berg, Alexander Kirillov. 2021

491 citations recorded by the source corpus.

We present Boundary IoU (Intersection-over-Union), a new segmentation evaluation measure focused on boundary quality. We perform an extensive analysis across different error types and object sizes and show that Boundary IoU is significantly more sensitive than the standard Mask IoU measure to boundary errors for large objects and does not over-penalize errors on smaller objects. The new quality measure displays several desirable characteristics like symmetry w.r.t. prediction/ground truth pairs and balanced responsiveness across scales, which makes it more suitable for segmentation evaluation than other boundary-focused measures like Trimap IoU and F-measure. Based on Boundary IoU, we update the standard evaluation protocols for instance and panoptic segmentation tasks by proposing the Boundary AP (Average Precision) and Boundary PQ (Panoptic Quality) metrics, respectively. Our experiments show that the new evaluation metrics track boundary quality improvements that are generally overlooked by current Mask IoU-based evaluation metrics. We hope that the adoption of the new boundary-sensitive evaluation metrics will lead to rapid progress in segmentation methods that improve boundary quality. 1

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Masked-attention Mask Transformer for Universal Image Segmentation (2021, methodology): “6 AP improvement over HTC++, the Boundary AP [12] improves by 2.”
Metrics Reloaded: Recommendations for image analysis validation (2022, unknown): “To this end, (1) the location of both the reference objects and the predicted objects must be adequately represented (for example, by masks, bounding boxes or center points), and (2) a metric for deciding on a match (for example, mask IoU) must be chosen.”
SOLQ: Segmenting Objects by Learning Queries (2021, methodology): “To further validate the quality of boundary prediction, we also evaluate SOLQ using the Boundary AP [55] in Appendix A.”
Learning to Upsample by Learning to Sample (2023, methodology): “Besides the commonly used mIoU metric, we also report the bIoU [6] metric to evaluate the boundary quality.”
Personalize Segment Anything Model with One Shot (2023, methodology): “The mIoU and bIoU (Cheng et al., 2021) are adopted for evaluation.”
A Metaverse: Taxonomy, Components, Applications, and Open Challenges (2022, unknown): “Cheng et al. [245] updated the standard evaluation protocol, for instance, and panoptic segmentation tasks by proposing Boundary AP (Average Precision) and Boundary PQ (Panoptic Quality) metrics, respectively, based on Boundary IoU for image segmentation evaluation.”
End-to-End Instance Edge Detection (2022, methodology): “To compensate for this, we use AP boundary [9] to evaluate our model when there is a mask output.”
LVIS Challenge Track Technical Report 1st Place Solution: Distribution Balanced and Boundary Refinement for Large Vocabulary Instance Segmentation (2021, background): “Therefore, the boundary AP evaluation metric [3] is proposed and puts forward higher requirements on the mask prediction quality.”
Segment Anything in High Quality (2023, methodology): “Evaluation Metrics To accurately quantify improvements in mask quality, instead of only employing the standard mask AP or mask mIoU, we also adopt boundary metrics mBIoU and boundary AP B [5].”