Instance-Aware Semantic Segmentation via Multi-task Network Cascades

By Jifeng Dai, Kaiming He. 2015

1,275 citations recorded by the source corpus.

Semantic segmentation research has recently witnessed rapid progress, but many leading methods are unable to identify object instances. In this paper, we present Multitask Network Cascades for instance-aware semantic segmentation. Our model consists of three networks, respectively differentiating instances, estimating masks, and categorizing objects. These networks form a cascaded structure, and are designed to share their convolutional features. We develop an algorithm for the nontrivial end-to-end training of this causal, cascaded structure. Our solution is a clean, single-step training framework and can be generalized to cascades that have more stages. We demonstrate state-of-the-art instance-aware semantic segmentation accuracy on PASCAL VOC. Meanwhile, our method takes only 360ms testing an image using VGG-16, which is two orders of magnitude faster than previous systems for this challenging problem. As a by product, our method also achieves compelling object detection results which surpass the competitive Fast/Faster R-CNN systems. The method described in this paper is the foundation of our submissions to the MS COCO 2015 segmentation competition, where we won the 1st place.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks (2015, methodology): “Since then, the frameworks of RPN and Faster R-CNN have been adopted and generalized to other methods, such as 3D object detection [13], part-based detection [14], instance segmentation [15], and image captioning [16].”
Mask R-CNN (2017, methodology): “We also compare to the RoIWarp operation proposed in [10].”
What makes ImageNet good for transfer learning? (2016, methodology): “Using ImageNet pre-trained CNN features, impressive results have been obtained on several image classification datasets [13, 35], object detection [16, 39], action recognition [41], human pose estimation [9], image segmentation [10], optical flow [47], image captioning [12, 22] and others [26].”
Semantic Understanding of Scenes Through the ADE20K Dataset (2016, methodology): “nt stuff and object segmentation is explored in [8] which uses precomputed superpixels and feature masking to represent stuff. Cascade of instance segmentation and categorization has been explored in [9]. A multiscale pyramid pooling module is proposed to improve the scene parsing [37]. A recent mul”
Cascade R-CNN: High Quality Object Detection and Instance Segmentation (2019, methodology): “Finally, [6] iterated between the detection and segmentation tasks, to achieve improved instance segmentation.”
Recurrent Neural Networks for Semantic Instance Segmentation (2017, methodology): “Most methods [17, 18, 24, 23, 11, 19] extend object detection pipelines based on object proposals (e.”
Fully Convolutional Instance-Aware Semantic Segmentation (2016, methodology): “For example, in the MNC method [8], which won the 1st place in COCO segmentation challenge 2015 [25], 10 layers in the ResNet-101 model [18] are kept in the per-ROI sub-network.”
Pixelwise Instance Segmentation with a Dynamically Instantiated Network (2017, methodology): “1https://github.com/daijifeng001/MNC”
Shape-aware Instance Segmentation (2016, methodology): “As in [7], we use an NMS threshold of 0.7, and ﬁnally keep the top 300 bounding box proposals.”
Deep Watershed Transform for Instance Segmentation (2016, methodology): “Current approaches generally use complex pipelines to handle instance extraction involving object proposals [20, 22, 7], conditional random fields (CRF) [32, 33], large recurrent neural networks (RNN) [24, 23, 2], or template matching [28].”
InstanceCut: From Edges to Instances with MultiCut (2016, methodology): “Several recent methods decompose the instance-aware segmentation problem into a detection stage and a foreground/background segmentation stage [16, 23].”
Instance-Sensitive Fully Convolutional Networks (2016, methodology): “In [20], the RPN boxes are used for regressing segmentation masks, conducted by an fc layer on Region-of-Interest (RoI) pooling features [21].”
Learning to Refine Object Segments (2016, methodology): “Hence, CNNs that utilize only upper network layers for object instance segmentation [21–23], as in Figure 1a, can eﬀectively generate coarse object masks but have diﬃculty generating pixel-accurate segmentations.”
MegDet: A Large Mini-Batch Object Detector (2017, methodology): “The improvements are mainly due to better backbone network [16], new detection framework [31], novel loss design [24], improved pooling method [5, 14], and so on [19].”
K-Net: Towards Unified Image Segmentation (2021, neutral): “‘Top-down’ approaches [2, 15, 22, 27, 33] first detect accurate”
Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation (2020, background): “Instance segmentation [22, 10] is an important task in computer vision with many real world applications.”
Cascade R-CNN: Delving Into High Quality Object Detection (2017, methodology): “[3] iterated a detection and a segmentation task alternatively, for instance segmentation.”
Scene Parsing through ADE20K Dataset (2017, background): “Cascade of instance segmentation and categorization has been explored in [9].”
Hybrid Task Cascade for Instance Segmentation (2019, neutral): “MNC [11] formulates instance segmentation as a pipeline that consists of three sub-tasks: instance localization, mask prediction and object categorization, and trains the whole network end-to-end in a cascaded manner.”