MaskLab: Instance Segmentation by Refining Object Detection with Semantic and Direction Features

By Liang-Chieh Chen, Alexander Hermans, George Papandreou, Florian Schroff, Peng Wang, Hartwig Adam. 2017

368 citations recorded by the source corpus.

In this work, we tackle the problem of instance segmentation, the task of simultaneously solving object detection and semantic segmentation. Towards this goal, we present a model, called MaskLab, which produces three outputs: box detection, semantic segmentation, and direction prediction. Building on top of the Faster-RCNN object detector, the predicted boxes provide accurate localization of object instances. Within each region of interest, MaskLab performs foreground/background segmentation by combining semantic and direction prediction. Semantic segmentation assists the model in distinguishing between objects of different semantic classes including background, while the direction prediction, estimating each pixel's direction towards its corresponding center, allows separating instances of the same semantic class. Moreover, we explore the effect of incorporating recent successful methods from both segmentation and detection (e.g., atrous convolution and hypercolumn). Our proposed model is evaluated on the COCO instance segmentation benchmark and shows comparable performance with other state-of-art models.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Stand-Alone Self-Attention in Vision Models (2019, neutral): “Given the success of content-based interactions on core computer vision tasks, we expect that future work may explore how attention could be applied to other vision tasks such as semantic segmentation [63], instance segmentation [64], keypoint detection [65], human pose estimation [66, 67] and other”
Panoptic Feature Pyramid Networks (2019, background): “While there have been attempts to unify semantic and instance segmentation [46, 1, 9], the specialization currently necessary to achieve top performance in each was perhaps inevitable given their parallel development and separate benchmarks.”
Hybrid Task Cascade for Instance Segmentation (2019, methodology): “MaskLab [7] produces instance-aware masks by combining semantic and direction predictions.”
Class-Incremental Instance Segmentation via Multi-Teacher Networks (2021, methodology): “One-stage methods (Dai et al. 2016; Liu et al. 2018; Bolya et al. 2019; Chen et al. 2018) are conceptually faster than the two-stage methods.”
Incremental Few-Shot Instance Segmentation (2021, neutral): “Approaches generally fall into two categories: grouping-based [4, 11, 17, 24] and proposal-based [2, 6, 15, 20] detection methods.”
Searching for Efficient Multi-Scale Architectures for Dense Image Prediction (2018, background): “and instance segmentation [20, 33, 9] rely on high resolution image inputs and multi-scale image representations.”
Mask Scoring R-CNN (2019, methodology): “[3] proposed MaskLab that used position-sensitive scores to obtain better results.”
Learning to Fuse Things and Stuff (2018, background): “[5] leverage semantic segmentation logits to refine instance segmentation masks, but not vice-versa.”
Meta R-CNN: Towards General Solver for Instance-Level Low-Shot Learning (2019, methodology): “Proposalbased methods [30, 31, 7, 6] predict object masks based on the generated region proposals while image-based methods [47, 48, 44, 2] produce a pixel-level segmentation map over the image to identify object instance.”
YOLACT: Real-Time Instance Segmentation (2019, methodology): “One-stage instance segmentation methods generate position sensitive maps that are assembled into final masks with position-sensitive pooling [6, 24] or combine semantic segmentation logits and direction prediction logits [4].”
InstaBoost: Boosting Instance Segmentation via Probability Map Guided Copy-Pasting (2019, methodology): “This idea is further developed by [9] which outperforms competing methods on the COCO dataset [33].”
SSAP: Single-Shot Instance Segmentation With Affinity Pyramid (2019, methodology): “Following works extend Mask R-CNN by modifying feature layers [38] or the mask prediction head [7].”
DeeperLab: Single-Shot Image Parser (2019, background): “The top-down approaches [57, 58, 59, 60, 61, 62, 63] obtain instance masks by refining the predicted boxes from state-of-the-art detectors [64, 65, 66].”
Affinity Derivation and Graph Merge for Instance Segmentation (2018, neutral): “MaskLab [6] combines Mask-RCNN with position-sensitive scores and shows an improvement on performance.”
Image Segmentation Using Deep Learning: A Survey (2020, methodology): “[66] developed an instance segmentation model, MaskLab, by refining object detection with semantic and direction features based on Faster R-CNN.”
AdaptIS: Adaptive Instance Selection Network (2019, background): “In recent years the approach to instance segmentation based on standard object detection pipeline [11, 1, 23] became a mainstream, achieving state-of-the-art results on various (a) (b) (c) (d) Figure 3.”
Learning to Manipulate Individual Objects in an Image (2020, background): “Semantic segmentation has improved considerably since the advent of deep neural networks [28, 8, 38] and so has instance segmentation, where the network has to distinguish between different instances of the same semantic class [31, 19, 27, 7].”
BlendMask: Top-Down Meets Bottom-Up for Instance Segmentation (2020, methodology): “This idea resembles MaskLab [5] and Instance Mask Projection (IMP) [10], which concatenates mask predictions with lower layers of backbone features.”
CentripetalNet: Pursuing High-Quality Keypoint Pairs for Object Detection (2020, neutral): “6 MaskLab+ [3] ResNet-101 37.”
DetectoRS: Detecting Objects with Recursive Feature Pyramid and Switchable Atrous Convolution (2020, methodology): “There are two main categories of object detection methods: one-stage methods, e.g ., [45,50,56,60, 80, 81], and multi-stage methods, e.g ., [5, 7, 9, 25, 27, 58].”