Structured Prediction Energy Networks

By David Belanger, Andrew McCallum. 2015

233 citations recorded by the source corpus.

We introduce structured prediction energy networks (SPENs), a flexible framework for structured prediction. A deep architecture is used to define an energy function of candidate labels, and then predictions are produced by using back-propagation to iteratively optimize the energy with respect to the labels. This deep architecture captures dependencies between labels that would lead to intractable graphical models, and performs structure learning by automatically learning discriminative features of the structured output. One natural application of our technique is multi-label classification, which traditionally has required strict prior assumptions about the interactions between labels to ensure tractable learning and prediction. We are able to apply SPENs to multi-label problems with substantially larger label sets than previous applications of structured prediction, while modeling high-order interactions using minimal structural assumptions. Overall, deep learning provides remarkable tools for learning features of the inputs to a prediction problem, and this work extends these techniques to learning features of structured outputs. Our experiments provide impressive performance on a variety of benchmark multi-label classification tasks, demonstrate that our technique can be used to provide interpretable structure learning, and illuminate fundamental trade-offs between feed-forward and iterative structured prediction.

Subfields: Artificial Intelligence, Materials Chemistry, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Materials Chemistry

In Model Architectures

Citations in the corpus

Structured Attention Networks (2017, methodology): “Many optimization algorithms (e.g. gradient descent, LBFGS) are also differentiable (Domke, 2012; Maclaurin et al., 2015), and have been used as output layers for structured prediction in energy-based models (Belanger & McCallum, 2016; Wang et al., 2016).”
Unrolled Generative Adversarial Networks (2016, background): “More recently, (Belanger & McCallum, 2015; Han et al., 2016; Andrychowicz et al., 2016) backpropagate through optimization procedures in contexts unrelated to GANs or minimax games.”
Structured Convolutional Kernel Networks for Airline Crew Scheduling (2021, methodology): “Furthermore, inspired by Q-learning, Gygli et al. (2017) used an oracle value function as the objective for energy-based deep networks, and Belanger & McCallum (2016) introduced structured prediction energy networks (SPENs) to address the inductive bias and to learn discriminative features of the…”
OptNet: Differentiable Optimization as a Layer in Neural Networks (2017, methodology): “…in this context requires observed data, which isn’t present in some of the contexts we consider in this paper, and also may suffer from instability issues when combined with deep energy-based architectures as observed in Belanger & McCallum (2016); Be-langer et al. (2017); Amos et al. (2017).”
Energy-based Latent Aligner for Incremental Learning (2022, methodology): “EBMs have been used for out-of-distribution sample detection [30, 47], structured prediction [4, 5, 48] and improving adversarial robustness [11, 17].”
A Distributional Approach to Controlled Text Generation (2020, background): “Some early NLP-related EBM research is concerned with neural-based sequence labelling problems (e.g. tagging) exploiting the global sequence (Andor et al., 2016; Belanger & McCallum, 2016).”
Input Convex Neural Networks (2016, methodology): “…to build general neural networks over joint input and output spaces, and perform inference over outputs using generic optimization techniques such as Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) and Structured Prediction Energy Networks (SPENs) (Belanger & McCallum, 2016).”
Column Networks for Collective Classification (2016, background): “There has been recent work that combines deep learning with structured prediction but the main learning and inference problems for general multi-relational settings remain open (Belanger and McCallum 2016; Do, Arti, and others 2010; Tompson et al. 2014; Yu, Wang, and Deng 2010; Zheng et al. 2015).”
Contextual Explanation Networks (2017, background): “…general representations (or embeddings) discovered by deep networks [8, 9, 10], representing potential functions [11] and energy functions [12] with neural networks, encoding learnable structure into Gaussian process kernels with deep and recurrent networks [13, 14], or learning…”
Energy-Based Models for Code Generation under Compilability Constraints (2021, background): “Early examples of EBMs applied to natu-ral language processing include sequence labeling problems (e.g. tagging) exploiting global properties of a sequence (Andor et al., 2016; Belanger and McCallum, 2016).”
Illuminating protein space with a programmable generative model (2022, background): “It is therefore natural to think about protein structure prediction as a so-called “Structured Prediction” problem [Belanger and McCallum, 2016] from machine learning, in which predictions are cast as the low-energy configurations of a learned potential function.”
Deep Value Networks Learn to Evaluate and Iteratively Refine Structured Outputs (2017, methodology): “Bibtex Bookmarks Logistic regression (Lin et al., 2014) 37.2 30.7 NN baseline (Belanger & McCallum, 2016) 38.9 33.8 SPEN (Belanger & McCallum, 2016) 42.2 34.4 PRLR (Lin et al., 2014) 44.2 34.9 DVN (Ours) 44.7 37.1 Table 1.”
Deep Structured Prediction with Nonlinear Output Transformations (2018, methodology): “SPENs: Most related to our approach is the recent work of Belanger and McCallum [3], Belanger et al. [4], which introduced structured prediction energy networks (SPENs) with the goal to address the inductive bias.”
Proximal Deep Structured Models (2016, result): “Our approach is also closely related to the recent structured prediction energy networks (SPEN) [?”
Detecting Visual Relationships with Deep Relational Networks (2017, background): “SPENs introduced in [57] define a neural network serving as an energy function over observed features for multi-label classification.”
Parametric Adversarial Divergences are Good Task Losses for Generative Modeling (2017, background): “Alternatively, the score function could also be a learned neural network (Belanger & McCallum, 2016).”
Distributional Reinforcement Learning for Energy-Based Sequential Models (2019, background): “[3] address a similar class of problems (multi-labelling problems such as sequence tagging), employing an energy-based generalization of CRFs, also focussing on inference as optimization.”
Deep Set Prediction Networks (2019, background): “This type of nested optimisation to obtain structured outputs with neural networks was first studied in [3].”