Efficient Piecewise Training of Deep Structured Models for Semantic Segmentation

By Guosheng Lin, Chunhua Shen, Anton van dan Hengel, Ian David Reid. 2015

940 citations recorded by the source corpus.

Recent advances in semantic image segmentation have mostly been achieved by training deep convolutional neural networks (CNNs). We show how to improve semantic segmentation through the use of contextual information, specifically, we explore 'patch-patch' context between image regions, and 'patch-background' context. For learning from the patch-patch context, we formulate Conditional Random Fields (CRFs) with CNN-based pairwise potential functions to capture semantic correlations between neighboring patches. Efficient piecewise training of the proposed deep structured model is then applied to avoid repeated expensive CRF inference for back propagation. For capturing the patch-background context, we show that a network design with traditional multi-scale image input and sliding pyramid pooling is effective for improving performance. Our experimental results set new state-of-the-art performance on a number of popular semantic segmentation datasets, including NYUDv2, PASCAL VOC 2012, PASCAL-Context, and SIFT-flow. In particular, we achieve an intersection-overunion score of 78:0 on the challenging PASCAL VOC 2012 dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Fully Convolutional Instance-Aware Semantic Segmentation (2016, background): “Recent works in [3, 43, 37, 24] integrated FCNs with conditional random fields (CRFs).”
Pixelwise Instance Segmentation with a Dynamically Instantiated Network (2017, neutral): “However, inference of this graphical model was not performed end-toend as shown to be possible in [53, 2, 4, 29].”
The Cityscapes Dataset for Semantic Urban Scene Understanding (2016, methodology): “Adelaide [37] was trained for 8 days using random crops of the input image as described above.”
DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs (2016, methodology): “official version Adelaide_Context [40] 71.”
Multi-Scale Context Aggregation by Dilated Convolutions (2015, background): “Another approach involves providing multiple rescaled versions of the image as input to the network and combining the predictions obtained for these multiple inputs (Farabet et al., 2013; Lin et al., 2015; Chen et al., 2015b).”
Pyramid Scene Parsing Network (2016, neutral): “5 Piecewise† [20] 94.”
Conditional Random Fields as Recurrent Neural Networks (2015, neutral): “Context-DeepCNN-CRF [33] n/a n/a 70.”
Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation (2018, background): “, image pyramid) [18, 16, 58, 44, 11, 9] or those that adopt probabilistic graphical models (such as DenseCRF [37] with efficient inference algorithm [2]) [8, 4, 82, 44, 48, 55, 63, 34, 72, 6, 7, 9].”
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation (2015, methodology): “They require either MAP inference over a CRF [42], [43] or aids such as region proposals [4] for inference.”
Semantic Image Segmentation via Deep Parsing Network (2015, methodology): “We evaluate DPN on several scales of the images and then average the results following [3, 19].”
UberNet: Training a Universal Convolutional Neural Network for Low-, Mid-, and High-Level Vision Using Diverse Datasets and Limited Memory (2016, background): “We leave for future work the inclusion of additional taskspecific convolutional filters [18, 42, 43, 49] or structured prediction operations [8, 10, 30, 37, 57].”
Large Kernel Matters — Improve Semantic Segmentation by Global Convolutional Network (2017, background): “Furthermore, Adelaide [21] deeply incorporates CRF and CNN where hand-crafted potentials is replaced by convolutions and nonlinearities.”
Proposal-Free Network for Instance-Level Object Segmentation (2015, neutral): “Index Terms—Instance-level Object Segmentation, Proposal-free, Convolutional Neural Network F”
Gather-Excite: Exploiting Feature Context in Convolutional Neural Networks (2018, background): “[48, 23]), and in particular, ParseNet [26] showed that encoding context through global feature averaging can be highly effective for this task.”