YOLACT: Real-Time Instance Segmentation

By Daniel Bolya, Chong Zhou, Fanyi Xiao, Yong Jae Lee. 2019

2,140 citations recorded by the source corpus.

We present a simple, fully-convolutional model for real-time instance segmentation that achieves 29.8 mAP on MS COCO at 33.5 fps evaluated on a single Titan Xp, which is significantly faster than any previous competitive approach. Moreover, we obtain this result after training on only one GPU. We accomplish this by breaking instance segmentation into two parallel subtasks: (1) generating a set of prototype masks and (2) predicting per-instance mask coefficients. Then we produce instance masks by linearly combining the prototypes with the mask coefficients. We find that because this process doesn't depend on repooling, this approach produces very high-quality masks and exhibits temporal stability for free. Furthermore, we analyze the emergent behavior of our prototypes and show they learn to localize instances on their own in a translation variant manner, despite being fully-convolutional. Finally, we also propose Fast NMS, a drop-in 12 ms faster replacement for standard NMS that only has a marginal performance penalty.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Generalized Decoding for Pixel, Image, and Language (2022, methodology): “Models for both tasks have evolved from CNNbased architectures [51] to transformer-based ones [11], and from two-stage models [29] to one-stage models [3, 71] and to the recent query-based approaches [18, 100].”
3D Instance Segmentation of MVS Buildings (2021, methodology): “Inspired by FCIS [28] and YOLACT [29], BlendMask uses a blender module to merge top-level coarse instance information with lower-level fine granularities [30].”
iShape: A First Step Towards Irregular Shape Instance Segmentation (2021, neutral): “Many methods [12, 13, 14, 15] and datasets [16, 17, 18] continue to emerge in this ﬁeld.”
K-Net: Towards Unified Image Segmentation (2021, neutral): “‘Top-down’ approaches [2, 15, 22, 27, 33] first detect accurate”
Class-Incremental Instance Segmentation via Multi-Teacher Networks (2021, methodology): “2017), we choose the one-stage method YOLACT (Bolya et al. 2019) as our incremental base instance segmentation model, because the prior boxes in YOLACT are agnostic to object categories.”
Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation (2019, background): “We note that there are several instance segmentation works [77, 68, 76, 2, 48, 35, 56, 20, 18, 44, 37, 30, 51, 54, 6], which could be potentially extended to bottom-up panoptic segmentation.”
Image Segmentation Using Deep Learning: A Survey (2020, background): “Another promising approach is to tackle the instance segmentation problem by learning grouping cues for bottom-up segmentation, such as deep watershed transform [73], real-time instance segmentation [74],”
PolarMask: Single Shot Instance Segmentation With Polar Representation (2019, background): “With a simpler pipeline and half training epochs, PolarMask outperforms YOLACT with 0.9 mAP.”
Spatial Feature Calibration and Temporal Fusion for Effective One-stage Video Instance Segmentation (2021, methodology): “One-stage instance segmentation approaches [42, 34, 7, 45, 38, 5, 40, 21] recently are proposed to keep the trade-off between speed and performance.”
DynaSLAM II: Tightly-Coupled Multi-Object Tracking and SLAM (2020, background): “Algorithms such as YOLACT [29] can run in real time and provide high-quality instance masks.”
Segment Everything Everywhere All at Once (2023, background): “On the other hand, instance segmentation involves grouping pixels that belong to the same semantic class into separate object instances [4, 43, 7].”
Deep Snake for Real-Time Instance Segmentation (2020, neutral): “YOLACT [3] ESE [42] OURS”
Enhancing Geometric Factors in Model Learning and Inference for Object Detection and Instance Segmentation (2020, methodology): “For instance segmentation, Mask R-CNN [21] adds an extra instance mask branch on Faster R-CNN, while the recent state-of-the-art YOLACT [20] does the same thing on RetinaNet [12].”
SOLOv2: Dynamic and Fast Instance Segmentation (2020, methodology): “To accelerate the inference, Fast NMS [2] enables deciding the predictions to be kept or discarded in parallel.”
Vision-based robotic grasping from object localization, object pose estimation to grasp estimation for parallel grippers: a review (2020, background): “, 2019c], YOLACT [Bolya et al., 2019b], YOLACT++ [Bolya et al.”
Multi-Task Collaborative Network for Joint Referring Expression Comprehension and Segmentation (2020, background): “Some recent works also focus on combining the object detection and segmentation into a joint framework, e.g. , MaskRCNN [10], YOLACT [2], and Reti-naMask [9].”
SOLO: Segmenting Objects by Locations (2019, neutral): “6 YOLACT [1] Res-101-FPN 31.”
Visible and Thermal Image-Based Trunk Detection with Deep Learning for Forestry Mobile Robotics (2021, neutral): “Lastly, a fast and simple model that was designed for real-time instance segmentation is YOLACT [70].”