Mask Scoring R-CNN

By Zhaojin Huang, Lichao Huang, Yongchao Gong, Chang Huang, Xinggang Wang. 2019

1,031 citations recorded by the source corpus.

Letting a deep network be aware of the quality of its own predictions is an interesting yet important problem. In the task of instance segmentation, the confidence of instance classification is used as mask quality score in most instance segmentation frameworks. However, the mask quality, quantified as the IoU between the instance mask and its ground truth, is usually not well correlated with classification score. In this paper, we study this problem and propose Mask Scoring R-CNN which contains a network block to learn the quality of the predicted instance masks. The proposed network block takes the instance feature and the corresponding predicted mask together to regress the mask IoU. The mask scoring strategy calibrates the misalignment between mask quality and mask score, and improves instance segmentation performance by prioritizing more accurate mask predictions during COCO AP evaluation. By extensive evaluations on the COCO dataset, Mask Scoring R-CNN brings consistent and noticeable gain with different models and outperforms the state-of-the-art Mask R-CNN. We hope our simple and effective approach will provide a new direction for improving instance segmentation. The source code of our method is available at \url{https://github.com/zjhuang22/maskscoring_rcnn}.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

3D Instance Segmentation of MVS Buildings (2021, methodology): “R-CNN first introduces CNN in the field of object detection [13].”
Learning Open-World Object Proposals without Learning to Classify (2021, methodology): “Researchers have studied many ways to improve the localization quality in object detections by learning centerness [14], iterative proposal refinement [5], or box/mask IoU prediction [15], [16], [36].”
K-Net: Towards Unified Image Segmentation (2021, background): “Extensions of this framework add a mask scoring branch [24] or adopt a cascade structure [3, 5].”
Occluded Video Instance Segmentation: A Benchmark (2021, background): “, 2015), instance segmentation (He et al., 2017; Huang et al., 2019; Kirillov et al., 2020), panoptic segmentation (Kirillov et al.”
Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection (2020, methodology): “the quality estimation is combined (usually multiplied) with classification confidence as final scores [12, 11, 26, 29, 35] f r the rank process of Non-Maximum Suppression (NMS) during inference.”
MMDetection: Open MMLab Detection Toolbox and Benchmark (2019, background): “• Mask Scoring R-CNN [15]: an improvement over Mask R-CNN by predicting the mask IoU, proposed in 2019.”
YOLACT: Real-Time Instance Segmentation (2019, background): “4 MS R-CNN [20] R-101-FPN 8.”
Conditional Convolutions for Instance Segmentation (2020, background): “Many works [7, 19, 26] with top performance are built on Mask R-CNN.”
Boundary IoU: Improving Object-Centric Image Segmentation Evaluation (2021, background): “, [36, 4, 1, 16, 21]), only a few address mask boundary quality.”
Camouflaged Object Detection (2020, methodology): “Compared with either the GOD [3,23,27,35,75,83] or the SOD [38,40,51,68,77, 78] models, SINet significantly decreases the training time (e.”
Crossover Learning for Fast Online Video Instance Segmentation (2021, background): “[28, 9, 13] promote Mask R-CNN and achieve better instance segmentation results.”
Bidirectional Graph Reasoning Network for Panoptic Segmentation (2020, methodology): “Mask Scoring R-CNN [16] corrected Mask R-CNN by aligning mask quality with mask score.”
PolarMask: Single Shot Instance Segmentation With Polar Representation (2019, background): “Mask R-CNN [15], built upon Faster R-CNN, simply adds an additional mask branch and use RoI-Align to replace RoI-Pooling [12] for improved accuracy.”
D2Det: Towards High Quality Object Detection and Instance Segmentation (2020, background): “Several recent works have extended this framework by, for example, integrating pyramid representations [31, 28, 44, 5], extending to multi-stage detection [16, 2, 6, 24] and integrating a mask branch [19, 23, 34].”
End-to-End Instance Edge Detection (2022, background): “Since then, researchers have explored various directions to improve efficiency [4,50] and effectiveness [29,31,13].”
The Devil is in Classification: A Simple Framework for Long-tail Instance Segmentation (2020, background): “For example, [20],[21] add IOU prediction branch to improve confidence scoring for object detection and instance segmentation respectively.”
Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers (2021, methodology): “BCGN achieves consistent improvement on different backbones and object detectors, demonstrating its effectiveness by outperforming both PANet [42] and Mask Scoring R-CNN [25] by 1.”
PolyTransform: Deep Polygon Transformer for Instance Segmentation (2019, positive): “[25] optimize the scoring of the bounding boxes by predicting IoU for each mask rather than only a classification score.”