Fully Connected Deep Structured Networks

By Alexander Gerhard Schwing, Raquel Urtasun. 2015

323 citations recorded by the source corpus.

Convolutional neural networks with many layers have recently been shown to achieve excellent results on many high-level tasks such as image classification, object detection and more recently also semantic segmentation. Particularly for semantic segmentation, a two-stage procedure is often employed. Hereby, convolutional networks are trained to provide good local pixel-wise features for the second step being traditionally a more global graphical model. In this work we unify this two-stage process into a single joint training algorithm. We demonstrate our method on the semantic image segmentation task and show encouraging results on the challenging PASCAL VOC 2012 dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Non-local Neural Networks (2017, background): “The iterative mean-field inference of CRF can be turned into a recurrent network and trained [56, 42, 8, 18, 34].”
Fully Convolutional Instance-Aware Semantic Segmentation (2016, background): “Recent works in [3, 43, 37, 24] integrated FCNs with conditional random fields (CRFs).”
InstanceCut: From Edges to Instances with MultiCut (2016, methodology): “of the methods utilize a conditional random field (CRF) model on top of an FCN [10, 36], or incorporate CRF-based mechanisms directly into a network architecture [39, 56, 45].”
The Cityscapes Dataset for Semantic Urban Scene Understanding (2016, methodology): “Following this line, [9, 37, 40, 63, 81] propose different approaches that combine the strengths of CNNs and Conditional Random Fields (CRFs).”
Learning to Refine Object Segments (2016, background): “Graphical model networks : a number of recent papers have proposed integrating graphical models into CNNs by demonstrating they can be formulated as recurrent nets [16–18].”
DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs (2016, methodology): “While we employ the CRF as a post-processing method, [40], [59], [62], [64], [65] have successfully pursued joint learning of the DCNN and CRF.”
Conditional Random Fields as Recurrent Neural Networks (2015, methodology): “emantic image segmentation which combines the strengths of deep learning and graphical modelling. Afterourinitialpublicationofthetechnicalreportofthis work on arXiv.org, a number of independent works [49, 35] appeared on arXiv.org presenting similar joint training approaches for semantic image segme”
Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation (2018, background): “, image pyramid) [18, 16, 58, 44, 11, 9] or those that adopt probabilistic graphical models (such as DenseCRF [37] with efficient inference algorithm [2]) [8, 4, 82, 44, 48, 55, 63, 34, 72, 6, 7, 9].”
Rethinking Atrous Convolution for Semantic Image Segmentation (2017, methodology): “2 for illustration) that exploit context information for semantic segmentation [25, 13, 52, 8, 81, 46, 55, 63, 75].”
ParseNet: Looking Wider to See Better (2015, methodology): “Starting from the FCN architecture, several approaches [2, 24, 18, 31], combining graphic model such as conditional random field (CRF), have been proposed to introduce global context and structured information into a FCN.”
Efficient Piecewise Training of Deep Structured Models for Semantic Segmentation (2015, methodology): “CRF-RNN [48] and the method in [40] extend DeepLab and [25] by jointly learning the dense CRFs and CNNs.”
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation (2015, methodology): “The fact that joint training helps is also shown in other recent results [41], [42].”
Semantic Image Segmentation via Deep Parsing Network (2015, methodology): “It has been adopted by most of the recent deep models [3, 39, 30] for semantic image segmentation.”
Searching for Efficient Multi-Scale Architectures for Dense Image Prediction (2018, background): “…models that contain encoder-decoder structures [2, 72, 49, 28, 64, 93, 96], or (3) designing models that employ a multi-scale context module, e.g ., DenseCRF module [42, 4, 10, 98, 50, 76], global context [56, 95], or atrous convolutions deployed in cascade [57, 94, 12] or in parallel [11, 12].”
Higher Order Conditional Random Fields in Deep Neural Networks (2015, methodology): “[4] showed further improvements by post-processing the results of a CNN with a CRF. Subsequent works [30, 34, 43, 53] have taken this idea further by incorporating a CRF as layers within a deep network and then learning parameters of both the CRF and CNN together via backpropagation.”
Structural-RNN: Deep Learning on Spatio-Temporal Graphs (2015, background): “Several works follow a two-stage approach and decouple the deep network from CRF.”
Monocular Object Instance Segmentation and Depth Ordering with CNNs (2015, methodology): “Data subsampling can be efficiently implemented for convolutions, which has been previously discussed in work by Mallat [27], known as the ‘à trous (with hole) algorithm’ and applied to CNNs in recent work [3, 29, 32].”
Semantic Image Segmentation with Task-Specific Edge Detection Using CNNs and a Discriminatively Trained Domain Transform (2015, methodology): “Further improvements have been shown in [46, 38] when backpropagating through the CRF to refine the segmentation CNN.”
Structured Prediction Energy Networks (2015, methodology): “It is natural to parametrize the potentials of a CRF using deep features (LeCun et al., 2006; Collobert et al., 2011; Jaderberg et al., 2014; Huang et al., 2015; Chen et al., 2014; Schwing & Urtasun, 2015; Chen et al., 2015).”