Bridging Category-level and Instance-level Semantic Image Segmentation

By Zifeng Wu, Chunhua Shen, Anton van den Hengel. 2016

176 citations recorded by the source corpus.

We propose an approach to instance-level image segmentation that is built on top of category-level segmentation. Specifically, for each pixel in a semantic category mask, its corresponding instance bounding box is predicted using a deep fully convolutional regression network. Thus it follows a different pipeline to the popular detect-then-segment approaches that first predict instances' bounding boxes, which are the current state-of-the-art in instance segmentation. We show that, by leveraging the strength of our state-of-the-art semantic segmentation models, the proposed method can achieve comparable or even better results to detect-then-segment approaches. We make the following contributions. (i) First, we propose a simple yet effective approach to semantic instance segmentation. (ii) Second, we propose an online bootstrapping method during training, which is critically important for achieving good performance for both semantic category segmentation and instance-level segmentation. (iii) As the performance of semantic category segmentation has a significant impact on the instance-level segmentation, which is the second step of our approach, we train fully convolutional residual networks to achieve the best semantic category segmentation accuracy. On the PASCAL VOC 2012 dataset, we obtain the currently best mean intersection-over-union score of 79.1%. (iv) We also achieve state-of-the-art results for instance-level segmentation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs (2016, neutral): “Our final model is slightly better than the concurrent work [93] by 1.”
Iteratively Trained Interactive Segmentation (2018, methodology): “For training our network, we use bootstrapped cross-entropy [27] as the loss function, which takes an average over the loss values at the pixels that represent the worst k predictions.”
Pyramid Scene Parsing Network (2016, methodology): “For fair comparison with current ResNet based frameworks [38, 9, 4] in scene parsing/semantic segmentation task, we build our architecture based on ResNet101 while without postprocessing like CRF.”
Hybrid Task Cascade for Instance Segmentation (2019, methodology): “Other approaches include bridging category-leval and instance-level segmentation [42], learning a boundary-aware mask representation [17], and employing a sequence of neural networks to deal with different sub-grouping problems [27].”
Wider or Deeper: Revisiting the ResNet Model for Visual Recognition (2016, methodology): “irectly apply the hyper-parameters which are used on method feature pixel acc. mean acc. mean IoU FCN-8s [22] VGG16 65.9 46.5 35.1 BoxSup [6] VGG16 – – 40.5 Context [18] VGG16 71.5 53.9 43.3 VeryDeep [34] ResNet-101 72.9 54.8 44.5 DeepLab-v2 [3] ResNet-101 – – 45.7 Model A2, 2 conv. 75.0 58.1 48.1 T”
Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation (2019, background): “The loss improves over bootstrapped cross entropy loss [72, 7, 59] by weighting each pixel differently.”
Context Encoding for Semantic Segmentation (2018, methodology): “BaseNet mIoU% FCN-8s [37] 37.8 CRF-RNN [60] 39.3 ParseNet [35] 40.4 BoxSup [9] 40.5 HO CRF [2] 41.3 Piecewise [32] 43.3 VeryDeep [51] 44.5 DeepLab-v2 [5] by “unique” operation ﬁnding the categories presented in the given ground-truth segmentation mask.”
MaskLab: Instance Segmentation by Refining Object Detection with Semantic and Direction Features (2017, background): “ reﬁne the results. Liu et al. [50] segment objects in multiscale patches and aggregate the results. Levinkov et al. [42] propose efﬁcient local search algorithms for instance segmentation. Wu et al. [72] exploit a localization network for grouping, while Bai and Urtasun [6] adopt a Watershed Transf”
Full-Resolution Residual Networks for Semantic Segmentation in Street Scenes (2016, methodology): “Following Wu et al ., we train our network by minimizing a bootstrapped cross-entropy loss [52].”
SPGNet: Semantic Prediction Guidance for Scene Parsing (2019, background): “We study the effects of on-line hard example (or pixel) mining (OHEM) [4, 63, 67] OHEM consistently improves the performance.”
DeeperLab: Single-Shot Image Parser (2019, methodology): “The semantic segmentation prediction is trained to minimize the bootstrapped cross-entropy loss [97, 98, 50], in which we sort the pixels based on the cross-entropy loss and we only backpropagate the errors in the top-K positions (hard example mining).”
Deep learning to achieve clinically applicable segmentation of head and neck anatomy for radiotherapy (2018, methodology): “We trained our network with a regularised top-k-percent pixel-wise binary cross-entropy loss [43]: for each output channel, we selected the top 5% most difﬁcult pixels (those with the highest binary cross-entropy) and only added their contribution to the total loss.”