Convolutional feature masking for joint object and stuff segmentation

By Jifeng Dai, Kaiming He. 2014

454 citations recorded by the source corpus.

The topic of semantic segmentation has witnessed considerable progress due to the powerful features learned by convolutional neural networks (CNNs) [13]. The current leading approaches for semantic segmentation exploit shape information by extracting CNN features from masked image regions. This strategy introduces artificial boundaries on the images and may impact the quality of the extracted features. Besides, the operations on the raw image domain require to compute thousands of networks on a single image, which is time-consuming. In this paper, we propose to exploit shape information via masking convolutional features. The proposal segments (e.g., super-pixels) are treated as masks on the convolutional feature maps. The CNN features of segments are directly masked out from these maps and used to train classifiers for recognition. We further propose a joint method to handle objects and “stuff” (e.g., grass, sky, water) in the same framework. State-of-the-art results are demonstrated on benchmarks of PASCAL VOC and new PASCAL-CONTEXT, with a compelling computational speed.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks (2015, background): “pooling (SPP) [1] on shared convolutional feature maps is developed for efficient region-based object detection [1], [30] and semantic segmentation [29].”
Object Detection Networks on Convolutional Feature Maps (2015, methodology): “n the SPP-based object detection method [13], features are pooled from proposal regions [27] on convolutional feature maps, and fed into the original fullyconnected layers for classifying regions. In [3], networks with a masking layer on feature maps are developed for semantic segmentation. The Full”
Fully convolutional networks for semantic segmentation (2014, methodology): “In Table 6, we compare to the joint object + stuff variation of Convolutional Feature Masking [4] which is the previous state-of-the-art on this task.”
Mask R-CNN (2017, methodology): “Earlier methods [13, 15, 16, 9] resorted to bottom-up segments [38, 2].”
Semantic Understanding of Scenes Through the ADE20K Dataset (2016, background): “Some of them [20, 1] are evaluated on Pascal Context [22] or SUN RGB-D dataset [31] to show the capability to segment more object classes in scenes. Joint stuff and object segmentation is explored in [8] which uses precomputed superpixels and feature masking to represent stuff. Cascade of instance s”
UPSNet: A Unified Panoptic Segmentation Network (2019, background): “Beginning with the introduction of region-based CNN (R-CNN) [12], many early deep learning approaches to instance segmentation attacked the problem by casting the solution to instance segmentation as a two stage approach where a number of segment proposals are made, which is then followed by a votin”
Learning Non-maximum Suppression (2017, methodology): “[37, 5] combine detections with semantic labelling, while [36] rephrase detection as a labelling problem.”
Fully Convolutional Instance-Aware Semantic Segmentation (2016, methodology): “In a prevalent family of instance-aware semantic segmentation approaches [7, 16, 8], it is achieved by adopting different types of sub-networks in three stages: 1) an FCN is applied on the whole image to generate intermediate and shared feature maps; 2) from the shared feature maps, a pooling layer ”
Hypercolumns for object segmentation and fine-grained localization (2014, methodology): “We also find that this architecture gives us the best performance on the SDS task so far: with simple bounding box detection followed by our hypercolumn-based mask prediction, we achieve a mean APr of 56.5 at an overlap threshold of 0.5 and a mean APr of 37.0 at an overlap threshold of 0.7.”
Pixelwise Instance Segmentation with a Dynamically Instantiated Network (2017, background): “However, approaches that segment instances by refining detections [18, 19, 7, 10, 26] are inherently limited by the quality of the initial proposals.”
Shape-aware Instance Segmentation (2016, methodology): “We used all the training images to learn our model, but, following the protocols used in [13, 14, 6, 7, 5], used only the validation dataset for evaluation.”
InstanceCut: From Edges to Instances with MultiCut (2016, background): “MCG [3] and then, in the second stage, a different network classifies these proposals [13, 22, 15, 11].”
The Cityscapes Dataset for Semantic Urban Scene Understanding (2016, methodology): “Following this line, [9, 37, 40, 63, 81] propose different approaches that combine the strengths of CNNs and Conditional Random Fields (CRFs).”
Instance-Aware Semantic Segmentation via Multi-task Network Cascades (2015, methodology): “All these methods [10, 13, 14, 7] rely on computationally expensive mask proposal methods.”
COCO-Stuff: Thing and Stuff Classes in Context (2016, background): “(5) Texture: Stuff classes are typically highly textured [21, 27, 51, 14].”
DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs (2016, background): “Further, [51] propose to pool the intermediate feature maps by region proposals.”
Learning Open-World Object Proposals without Learning to Classify (2021, background): “They have become the integral pre-processing steps for many computer vision systems, including object detection [33,41,42,45,59,74], segmentation [3,8,13], object discovery [12, 16, 47], weakly supervised object detection [5, 21, 53], visual tracking [35, 63], content-aware retargeting [51], etc.”