Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs

By Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin P. Murphy, Alan L. Yuille. 2014

5,264 citations recorded by the source corpus.

Deep Convolutional Neural Networks (DCNNs) have recently shown state of the art performance in high level vision tasks, such as image classification and object detection. This work brings together methods from DCNNs and probabilistic graphical models for addressing the task of pixel-level classification (also called "semantic image segmentation"). We show that responses at the final layer of DCNNs are not sufficiently localized for accurate object segmentation. This is due to the very invariance properties that make DCNNs good for high level tasks. We overcome this poor localization property of deep networks by combining the responses at the final DCNN layer with a fully connected Conditional Random Field (CRF). Qualitatively, our "DeepLab" system is able to localize segment boundaries at a level of accuracy which is beyond previous methods. Quantitatively, our method sets the new state-of-art at the PASCAL VOC-2012 semantic image segmentation task, reaching 71.6% IOU accuracy in the test set. We show how these results can be obtained efficiently: Careful network re-purposing and a novel application of the 'hole' algorithm from the wavelet community allow dense computation of neural net responses at 8 frames per second on a modern GPU.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Neural Machine Translation in Linear Time (2016, methodology): “The masked convolutions use dilation to increase the receptive ﬁeld of the target network (Chen et al., 2014; Yu & Koltun, 2015).”
Object Detection Networks on Convolutional Feature Maps (2015, methodology): “Then we adopt the “hole algorithm” [33], 464 [34] (“Algorithme à trous” [35]) on all following layers 465 to compensate this modification.”
Fully convolutional networks for semantic segmentation (2014, background): “ervised semantic segmentation [35], [36], [37], [38]. In addition, new works have improved the FCNs presented here to further advance the state-of-the-art in semantic segmentation. The DeepLab models [39] raise output resolution by dilated convolution and dense CRF inference. The joint CRFasRNN [40]”
Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, methodology): “Some employ atrous convolutions [34,77,67,13], larger kernel [70], or image pyramids [98,85], either designed by hand or searched by algorithms [103,12,60].”
Weight Standardization (2019, background): “Deep neural networks achieve state-of-the-arts in many vision tasks [4, 14, 20, 25, 31, 32, 39, 46].”
CCNet: Criss-Cross Attention for Semantic Segmentation (2018, methodology): “…Structure For semantic segmentation, we choose the ImageNet pre-trained ResNet-101 [17] as our backbone and remove the last two down-sampling operations and employ dilated convolutions in the subsequent convolutional layers following the previous work [4], resulting in the output stride as 8.”
Non-local Neural Networks (2017, neutral): “In the context of deep neural networks, a CRF can be exploited to post-process semantic segmentation predictions of a network [9].”
Making Convolutional Networks Shift-Invariant Again (2019, methodology): “Deep networks can also be evaluated densely, by removing striding and making appropriate changes to subsequent layers by using á trous/dilated convolutions (Chen et al., 2015; 2018; Yu & Koltun, 2016; Yu et al., 2017).”
Unified Perceptual Parsing for Scene Understanding (2018, methodology): “Pioneering work by Chen et al. [10], based on structure prediction, uses conditional random ﬁeld (CRF) to reﬁne the activations of the ﬁnal feature map of CNNs.”
UPSNet: A Unified Panoptic Segmentation Network (2019, background): “Recent semantic segmentation methods that exploit deep convolutional feature extraction mainly approach this problem from either multi-scale feature aggregation [40, 26, 5, 14], or end-to-end structured prediction [2, 41, 1, 25, 4] perspectives.”
DeepSetNet: Predicting Sets with Deep Neural Networks (2016, background): “Deep convolutional (CNN) and recurrent (RNN) neural networks now outperform traditional approaches in tasks like semantic segmentation [6, 25], image captioning [21] or object detection [27].”
Fully Convolutional Instance-Aware Semantic Segmentation (2016, methodology): “Recent works in [3, 43, 37, 24] integrated FCNs with conditional random fields (CRFs).”
SSD: Single Shot MultiBox Detector (2015, methodology): “3, we used the atrous version of VGG16, following DeepLab-LargeFOV [16].”
Recurrent Instance Segmentation (2015, methodology): “Semantic segmentation methods have seen a significant improvement recently, based first on the work in [12] which proposed a fully convolutional network that produces pixel-wise predictions, followed by the works in [13, 14] that improve the delineation of the predicted objects by using Conditional ”
Pixelwise Instance Segmentation with a Dynamically Instantiated Network (2017, methodology): “[28] developed another proposal-free method based on the semantic segmentation network of [5].”
Beyond Skip Connections: Top-Down Modulation for Object Detection (2016, methodology): “After the resurgence of ConvNets for image classification [8, 29], they have been successfully adopted for a variety of computer vision tasks such as object detection [16, 17, 41, 48], semantic segmentation [3, 6, 34, 35], instance segmentation [21, 22, 38], pose estimation [49, 50], depth estimatio”
Shape-aware Instance Segmentation (2016, background): “Over the years, much progress has been made on the task of category-level semantic segmentation, particularly since the advent of Deep Convolutional Neural Networks (CNNs) [9, 22, 3].”
R-FCN: Object Detection via Region-based Fully Convolutional Networks (2016, methodology): “…and on the conv 4 stage [10] (stride=16) are unchanged; the stride=2 operations in the ﬁrst conv 5 block is modiﬁed to have stride=1, and all convolutional ﬁlters on the conv 5 stage are modiﬁed by the “hole algorithm” [16, 2] (“ Algorithme à trous ” [17]) to compensate for the reduced stride.”
The Cityscapes Dataset for Semantic Urban Scene Understanding (2016, methodology): “(6) Coarse labels can complement the dense labels if applying appropriate methods as evidenced by [48] outperforming [9], which it extends by exploiting both dense and weak annotations (e.”