YOLACT++ Better Real-Time Instance Segmentation

By Daniel Bolya, Chong Zhou, Fanyi Xiao, Yong Jae Lee. 2019

506 citations recorded by the source corpus.

We present a simple, fully-convolutional model for real-time (<inline-formula><tex-math notation="LaTeX">$>30$</tex-math><alternatives><mml:math><mml:mrow><mml:mo>></mml:mo><mml:mn>30</mml:mn></mml:mrow></mml:math><inline-graphic xlink:href="zhou-ieq1-3014297.gif"/></alternatives></inline-formula> fps) instance segmentation that achieves competitive results on MS COCO evaluated on a single Titan Xp, which is significantly faster than any previous state-of-the-art approach. Moreover, we obtain this result after training on <italic>only one GPU</italic>. We accomplish this by breaking instance segmentation into two parallel subtasks: (1) generating a set of prototype masks and (2) predicting per-instance mask coefficients. Then we produce instance masks by linearly combining the prototypes with the mask coefficients. We find that because this process doesn’t depend on repooling, this approach produces very high-quality masks and exhibits temporal stability for free. Furthermore, we analyze the emergent behavior of our prototypes and show they learn to localize instances on their own in a translation variant manner, despite being fully-convolutional. We also propose Fast NMS, a drop-in 12 ms faster replacement for standard NMS that only has a marginal performance penalty. Finally, by incorporating deformable convolutions into the backbone network, optimizing the prediction head with better anchor scales and aspect ratios, and adding a novel fast mask re-scoring branch, our YOLACT++ model can achieve 34.1 mAP on MS COCO at 33.5 fps, which is fairly close to the state-of-the-art approaches while still running at real-time.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Instance Segmentation for Autonomous Log Grasping in Forestry Operations (2022, background): “One example is YOLACT++ [11], which is extending RetinaNet’s [12] dense grid prediction by adding a mask branch that combines prototype masks extracted from the feature maps.”
Masked-attention Mask Transformer for Universal Image Segmentation (2021, methodology): “, using dynamic kernels [3, 49, 56] or clustering algorithms [11, 29].”
Instances as Queries (2021, unknown): “In addition to R-CNN based methods, YOLACT [3, 2], SipMask [6], CondInst [46] and SOLO [51, 52] build one-stage instance segmentation framework on the top of one-stage framework, achieving comparable results with favorable inference speed.”
Spatial Feature Calibration and Temporal Fusion for Effective One-stage Video Instance Segmentation (2021, methodology): “Note that all experiments do not include other improvements in Yolact++ [4] such as more anchors and semantic segmentation loss.”
BlendMask: Top-Down Meets Bottom-Up for Instance Segmentation (2020, methodology): “‘+deform convs (interval = 3)’ uses deformable convolution in the backbone with interval 3, following [2].”
Vision-based robotic grasping from object localization, object pose estimation to grasp estimation for parallel grippers: a review (2020, methodology): “YOLACT is the first real-time one-stage instance segmentation method and is improved by YOLACT++ Bolya et al. (2019). PolarMask Xie et al. (2020) formulates the instance segmentation problem as predicting contour of instance through instance center classification and dense distance regression in a p”
SipMask: Spatial Information Preservation for Fast Image and Video Instance Segmentation (2020, methodology): “A recent variant of YOLACT, called YOLACT++ [3], utilizes a deformable backbone (ResNet101-Deform [55] with interval 3) and a mask scoring strategy.”
ST3D: Self-training for Unsupervised Domain Adaptation on 3D Object Detection (2021, unknown): “Though IoU regression has been tried to improve supervised image object detection performance [18, 3], to the best of our knowledge, we are the first to demonstrate that it can serve as a good criterion to assess the quality of pseudo box for UDA self-training with encouraging results.”
Interactive Object Segmentation With Inside-Outside Guidance (2022, methodology): “One-stage instance segmentation methods such as [26], [63], [64], [65], [66] generate position sensitive maps to acquire the ﬁnal segmentation mask.”
Crossover Learning for Fast Online Video Instance Segmentation (2021, methodology): “Recently, instance segmentation methods based on one-stage frameworks without explicit feature alignment operation begin to emerge [5, 4, 8, 65, 58, 59].”
TIDE: A General Toolbox for Identifying Object Detection Errors (2020, methodology): “Distinct from the two-stage R-CNN style approaches, we also include three single-stage approaches, YOLACT/YOLACT++ [3,2] to represent real-time models, RetinaNet [19] as a strong anchor-based model, Fully Convolutional One-Stage Object Detection (FCOS) [23] as a non anchor-based approach.”
Learning What Not to Segment: A New Perspective on Few-Shot Segmentation (2022, background): “However, collecting sufficient labeled data is notoriously time-consuming and labor-intensive, especially for dense prediction tasks, such as instance segmentation [2,3,15,21,59] and semantic segmentation [1,25,35,40,45].”
Box-supervised Instance Segmentation with Level Set Evolution (2022, background): “Another category [4, 5, 43, 48, 51] directly segments each instance in a fully convolutional manner without resorting to the detection results.”
DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision (2021, methodology): “Our best model achieves 37.9% AP on COCO test-dev, surpassing competitive supervised methods such as YOLACT++ [50] (34.6% AP) and Mask R-CNN [51] (37.1% AP).”