End-to-End Learning for Structured Prediction Energy Networks

By David Belanger, Bishan Yang, Andrew McCallum. 2017

138 citations recorded by the source corpus.

Structured Prediction Energy Networks (SPENs) are a simple, yet expressive family of structured prediction models (Belanger and McCallum, 2016). An energy function over candidate structured outputs is given by a deep network, and predictions are formed by gradient-based optimization. This paper presents end-to-end learning for SPENs, where the energy function is discriminatively trained by back-propagating through gradient-based prediction. In our experience, the approach is substantially more accurate than the structured SVM method of Belanger and McCallum (2016), as it allows us to use more sophisticated non-convex energies. We provide a collection of techniques for improving the speed, accuracy, and memory requirements of end-to-end SPENs, and demonstrate the power of our method on 7-Scenes image denoising and CoNLL-2005 semantic role labeling tasks. In both, inexact minimization of non-convex SPEN energies is superior to baseline methods that use simplistic energy functions that can be minimized exactly.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation (2021, background): “Problems in DSPN. Increasing the number of optimization steps for solving Equation 7 generally results in a better solution (Zhang et al., 2019), but requires signiﬁcantly more memory and computation time and can lead to training problems (Belanger et al., 2017).”
OptNet: Differentiable Optimization as a Layer in Neural Networks (2017, methodology): “Among many other applications, these architectures are well-studied for generic classification and structured prediction tasks (Goodfellow et al., 2013; Stoyanov et al., 2011; Brakel et al., 2013; LeCun et al., 2006; Belanger & McCallum, 2016; Belanger et al., 2017; Amos et al., 2017); in vision for”
Melding the Data-Decisions Pipeline: Decision-Focused Learning for Combinatorial Optimization (2018, background): “Our work is also related to recent research in structured prediction (Belanger, Yang, and McCallum 2017; Tu and Gimpel 2018; Niculae et al. 2018; Djolonga and Krause 2017). which aims to make a prediction lying in a discrete set.”
Deep Value Networks Learn to Evaluate and Iteratively Refine Structured Outputs (2017, background): “Concurrent to our work, (Belanger et al., 2017) explored another way of improving the training of SPENs, by directly back-Deep propagating the error through the gradient-based inference process.”
Semi-Amortized Variational Autoencoders (2018, methodology): “We propose an approach that leverages differentiable optimization (Domke, 2012; Maclaurin et al., 2015; Belanger et al., 2017) and differentiates through SVI while training the inference network/generative model.”
Deep Set Prediction Networks (2019, methodology): “That is, we use the average set loss 1 T ∑ t Lset(Ŷ ,Y ) as loss (similar to [4]).”
Concept Learning with Energy-Based Models (2018, methodology): “This form is similar to contrastive divergence [16] and structured SVM forms [2] and is a special case of guided cost learning formulation [8].”
RNA Secondary Structure Prediction By Learning Unrolled Algorithms (2020, background): “Previous works (Hershey et al., 2014; Belanger et al., 2017; Ingraham et al., 2018) did not discuss these aspects.”
Learning Approximate Inference Networks for Structured Prediction (2018, methodology): “Belanger et al. (2017) develop an “end-to-end” method that unrolls an approximate energy minimization algorithm into a ﬁxed-size computation graph that is trainable by gradient descent.”
Deep Structured Energy-Based Image Inpainting (2018, methodology): “[20] introduced an end-to-end learning method for SPENs, where an energy function is minimized by back-propagation using gradient-based prediction.”
SparseMAP: Differentiable Sparse Structured Inference (2018, background): “, 2011; Domke, 2013) and gradient descent (Belanger et al., 2017), where the backward pass complexity scales with the number of iterations.”
General Multi-label Image Classification with Transformers (2020, background): “(3) Structured Output: The third kind describes label dependencies using structured output inference formulation [29, 47, 2, 20, 32, 33, 56, 38].”
Differentiable Convex Optimization Layers (2019, methodology): “If the cone program is not differentiable at a solution, we compute a heuristic quantity, as is common practice in automatic differentiation [46, §14].”
Set Prediction without Imposing Structure as Conditional Density Estimation (2020, methodology): “However, they obtain predictions by minimizing the energy via (deterministic) gradient descent and require memory-intensive backpropagation through the unrolled inner optimization during training (Domke, 2012; Belanger et al., 2017).”
Deep Structured Prediction with Nonlinear Output Transformations (2018, background): “For example, it is reported that SPENs are hard to optimize [4].”
Towards Decoding as Continuous Optimisation in Neural Machine Translation (2017, methodology): “Recently, relaxation techniques have been applied to deep models for training and inference in text classiﬁcation (Belanger and McCallum, 2016; Belanger et al., 2017), and fully differentiable training of sequence-to-sequence models with scheduled-sampling (Goyal et al., 2017).”