Semantic Image Segmentation via Deep Parsing Network

By Ziwei Liu, Xiaoxiao Li, Ping Luo, Chen Change Loy, Xiaoou Tang. 2015

671 citations recorded by the source corpus.

This paper addresses semantic image segmentation by incorporating rich information into Markov Random Field (MRF), including high-order relations and mixture of label contexts. Unlike previous works that optimized MRFs using iterative algorithm, we solve MRF by proposing a Convolutional Neural Network (CNN), namely Deep Parsing Network (DPN), which enables deterministic end-to-end computation in a single forward pass. Specifically, DPN extends a contemporary CNN architecture to model unary terms and additional layers are carefully devised to approximate the mean field algorithm (MF) for pairwise terms. It has several appealing properties. First, different from the recent works that combined CNN and MRF, where many iterations of MF were required for each training image during back-propagation, DPN is able to achieve high performance by approximating one iteration of MF. Second, DPN represents various types of pairwise terms, making many existing works as its special cases. Third, DPN makes MF easier to be parallelized and speeded up in Graphical Processing Unit (GPU). DPN is thoroughly evaluated on the PASCAL VOC 2012 dataset, where a single DPN model yields a new state-of-the-art segmentation accuracy of 77.5%.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

CCNet: Criss-Cross Attention for Semantic Segmentation (2018, methodology): “Conditional random ﬁeld (CRF) [4, 43], Markov random ﬁeld (MRF) [25] are also utilized to capture long-range dependencies for semantic segmentation.”
UPSNet: A Unified Panoptic Segmentation Network (2019, background): “Recent semantic segmentation methods that exploit deep convolutional feature extraction mainly approach this problem from either multi-scale feature aggregation [40, 26, 5, 14], or end-to-end structured prediction [2, 41, 1, 25, 4] perspectives.”
InstanceCut: From Edges to Instances with MultiCut (2016, methodology): “of the methods utilize a conditional random field (CRF) model on top of an FCN [10, 36], or incorporate CRF-based mechanisms directly into a network architecture [39, 56, 45].”
The Cityscapes Dataset for Semantic Urban Scene Understanding (2016, methodology): “DPN [40] was trained using the original procedure, while using all available Cityscapes annotations.”
DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs (2016, methodology): “This direction has been extended by several follow-up papers [17], [40], [58], [59], [60], [61], [62], [63], [65], since the first version of our work was published [38].”
Pyramid Scene Parsing Network (2016, methodology): “Following methods [25, 41, 1] refined networks via end-to-end modeling.”
Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation (2018, background): “, image pyramid) [18, 16, 58, 44, 11, 9] or those that adopt probabilistic graphical models (such as DenseCRF [37] with efficient inference algorithm [2]) [8, 4, 82, 44, 48, 55, 63, 34, 72, 6, 7, 9].”
Rethinking Atrous Convolution for Semantic Image Segmentation (2017, methodology): “Furthermore, [81, 46, 63] propose to jointly train both the CRF and DCNN components, while [50, 78] employ several extra convolutional layers on top of the belief maps of DCNNs (belief maps are the final DCNN feature maps that contain output channels equal to the number of predicted classes) to capt”
Hybrid Task Cascade for Instance Segmentation (2019, background): “This task is closely related to both object detection and semantic segmentation [30, 22].”
Efficient Piecewise Training of Deep Structured Models for Semantic Segmentation (2015, methodology): “Using VOC+COCO training set, our method outperforms DPN [31] on 15 categories out of 20.”
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation (2015, background): “Other challenges, such as Pascal VOC12 [20] salient object segmentation have occupied researchers more [65], but we believe indoor scene segmentation is more challenging and has more current practical applications such as in AR and robotics.”
Dual Attention Network for Scene Segmentation (2018, methodology): “The work (Liu et al. 2015) adopts additional convolution layers to approximate the mean ﬁeld algorithm (MF) for capturing high-order relations.”
Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers (2020, background): “While the predictions of FCN are relatively coarse, several CRF/MRF [6, 35, 62] based approaches are developed to help reﬁne the coarse predictions.”
RefineNet: Multi-path Refinement Networks for High-Resolution Semantic Segmentation (2016, methodology): “Following the common convention [5, 6, 47, 35], the training set is augmented by additional annotated VOC images provided in [21] as well as with the training data from the MS COCO dataset [31].”
Wider or Deeper: Revisiting the ResNet Model for Visual Recognition (2016, neutral): “ 80.8 84.4 30.4 78.2 60.4 80.5 77.8 83.1 80.6 59.5 82.8 47.8 78.3 67.1 72.0 DeconvNet [26] 89.9 39.3 79.7 63.9 68.2 87.4 81.2 86.1 28.5 77.0 62.0 79.0 80.3 83.6 80.2 58.8 83.4 54.3 80.7 65.0 72.5 DPN [21] 87.7 59.4 78.4 64.9 70.3 89.3 83.5 86.1 31.7 79.9 62.6 81.9 80.0 83.5 82.3 60.5 83.2 53.4 77.9 ”
Higher Order Conditional Random Fields in Deep Neural Networks (2015, methodology): “[4] showed further improvements by post-processing the results of a CNN with a CRF. Subsequent works [30, 34, 43, 53] have taken this idea further by incorporating a CRF as layers within a deep network and then learning parameters of both the CRF and CNN together via backpropagation.”