Learning Deep Structured Models

By Liang-Chieh Chen, Alexander Gerhard Schwing, Alan L. Yuille, Raquel Urtasun. 2014

251 citations recorded by the source corpus.

Many problems in real-world applications involve predicting several random variables that are statistically related. Markov random fields (MRFs) are a great mathematical tool to encode such dependencies. The goal of this paper is to combine MRFs with deep learning to estimate complex representations while taking into account the dependencies between the output random variables. Towards this goal, we propose a training algorithm that is able to learn structured models jointly with deep features that form the MRF potentials. Our approach is efficient as it blends learning and inference and makes use of GPU acceleration. We demonstrate the effectiveness of our algorithm in the tasks of predicting words from noisy images, as well as tagging of Flickr photographs. We show that joint learning of the deep features and the MRF parameters results in significant performance gains.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Structured Attention Networks (2017, background): “…applications, most notably in seminal work on graph transformers (LeCun et al., 1998), key work on NLP (Collobert et al., 2011), and in many other areas (Peng et al., 2009; Do & Arti´eres, 2010; Jaderberg et al., 2014; Chen et al., 2015; Durrett & Klein, 2015; Lample et al., 2016, inter alia ).”
Pixelwise Instance Segmentation with a Dynamically Instantiated Network (2017, neutral): “However, inference of this graphical model was not performed end-toend as shown to be possible in [53, 2, 4, 29].”
Discriminative Embeddings of Latent Variable Models for Structured Data (2016, neutral): “In these existing messages learning work (Hershey et al., 2014; Zheng et al., 2015; Lin et al., 2015; Chen et al., 2014), the learning task is still estimating the messages represented graphical models with designed function forms, e.g., RNN or CNN, by maximizing loglikelihood.”
DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs (2016, methodology): “While we employ the CRF as a post-processing method, [40], [59], [62], [64], [65] have successfully pursued joint learning of the DCNN and CRF.”
Structured Convolutional Kernel Networks for Airline Crew Scheduling (2021, methodology): “, 2015a), and deep structured models (Chen et al., 2015)), it has far fewer parameters, thus bypassing the need for extensive tuning.”
Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs (2014, methodology): “There are multiple aspects in our model that we intend to reﬁne, such as fully integrating its two main components (CNN and CRF) and train the whole system in an end-to-end fashion, similar to Kr¨ahenb¨uhl & Koltun (2013); Chen et al. (2014); Zheng et al. (2015).”
Conditional Random Fields as Recurrent Neural Networks (2015, methodology): “[11] focused on the image classification problem with a similar approach.”
Efficient Piecewise Training of Deep Structured Models for Semantic Segmentation (2015, background): “The authors of [4] explore joint training of Markov random fields and deep neural networks for predicting words from noisy images and image s classification.”
Fully Connected Deep Structured Networks (2015, methodology): “Not only does the approach presented by [4] fail if the decomposition assumed in Eq.”
Attention to Scale: Scale-Aware Semantic Image Segmentation (2015, background): “Motivated by [6, 33, 50, 54], we further introduce extra supervision to the output of FCNs at each scale, which we find essential for a better performance.”
Integrating Learning and Reasoning with Deep Logic Models (2019, methodology): “Deep Structured Models [6, 19] use a similar graphical model to bridge the sensorial and semantic levels.”
OptNet: Differentiable Optimization as a Layer in Neural Networks (2017, methodology): “In recent years, there has been a strong push to further incorporate structured prediction methods like conditional random fields as the “last layer” of a deep network architecture (Peng et al., 2009; Zheng et al., 2015; Chen et al., 2015) as well as in deeper energy-based architectures (Belanger & ”
Human Pose Estimation with Iterative Error Feedback (2015, background): “see references in [26]) but it is a relatively modern topic in conjunction with feature learning, for computer vision [4, 18, 37, 23].”
Weakly-and Semi-Supervised Learning of a Deep Convolutional Network for Semantic Image Segmentation (2015, background): “Additional gains can be obtained by integrated end-to-end training of the DCNN and CRF parameters [41, 6].”
Structural-RNN: Deep Learning on Spatio-Temporal Graphs (2015, methodology): “[7] use a similar approach for image classification with general MRF.”
Semantic Image Segmentation with Task-Specific Edge Detection Using CNNs and a Discriminatively Trained Domain Transform (2015, background): “This allows us to use the DT as a layer in a CNN, thereby allowing us to jointly learn DCNNs that compute the coarse segmentation score maps in x and the reference edge map in g.”
Structured Prediction Energy Networks (2015, methodology): “It is natural to parametrize the potentials of a CRF using deep features (LeCun et al., 2006; Collobert et al., 2011; Jaderberg et al., 2014; Huang et al., 2015; Chen et al., 2014; Schwing & Urtasun, 2015; Chen et al., 2015).”
Fast, Exact and Multi-scale Inference for Semantic Image Segmentation with Deep Gaussian CRFs (2016, methodology): “A dominant research direction for improving semantic segmentation with deep learning is the combination of the powerful classification capabilities of FCNs with structured prediction [1,2,3,9,10,11], which aims at improving classification by capturing interactions between predicted labels.”
Input Convex Neural Networks (2016, background): “In recent years, there has been a strong push to further incorporate structured prediction methods like conditional random ﬁelds as the “last layer” of a deep network architecture (Peng et al., 2009; Zheng et al., 2015; Chen et al., 2015).”
Adversarial Diversity and Hard Positive Generation (2016, background): “Deep neural networks are powerful learning models which have been successfully applied to vision, speech and many other tasks [9, 11, 18, 19, 10, 20, 4, 3].”