Learning Approximate Inference Networks for Structured Prediction

By Lifu Tu, Kevin Gimpel. 2018

54 citations recorded by the source corpus.

Structured prediction energy networks (SPENs; Belanger & McCallum 2016) use neural network architectures to define energy functions that can capture arbitrary dependencies among parts of structured outputs. Prior work used gradient descent for inference, relaxing the structured output to a set of continuous variables and then optimizing the energy with respect to them. We replace this use of gradient descent with a neural network trained to approximate structured argmax inference. This "inference network" outputs continuous values that we treat as the output structure. We develop large-margin training criteria for joint training of the structured energy function and inference network. On multi-label classification we report speed-ups of 10-60x compared to (Belanger et al, 2017) while also improving accuracy. For sequence labeling with simple structured energies, our approach performs comparably to exact inference while being much faster at test time. We then demonstrate improved accuracy by augmenting the energy with a "label language model" that scores entire output label sequences, showing it can improve handling of long-distance dependencies in part-of-speech tagging. Finally, we show how inference networks can replace dynamic programming for test-time inference in conditional random fields, suggestive for their general use for fast inference in structured settings.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Energy-based Latent Aligner for Incremental Learning (2022, methodology): “EBMs have been used for out-of-distribution sample detection [30, 47], structured prediction [4, 5, 48] and improving adversarial robustness [11, 17].”
Neural Message Passing for Multi-Label Classification (2019, methodology): “To overcome the naive pairwise dependency constraint of CRFs, structured prediction energy networks (SPENS) [5] and related methods [50, 20] locally optimize an unconstrained structured output.”
ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translation (2020, methodology): “…networks to output approximate minimizers of a loss function is popular in variational inference (Kingma and Welling, 2013; Rezende et al., 2014), and, more recently, in structured prediction (Tu and Gimpel, 2018, 2019; Tu et al., 2019), including previously for neural MT (Gu et al., 2018b).”
Melding the Data-Decisions Pipeline: Decision-Focused Learning for Combinatorial Optimization (2018, background): “Our work is also related to recent research in structured prediction (Belanger, Yang, and McCallum 2017; Tu and Gimpel 2018; Niculae et al. 2018; Djolonga and Krause 2017). which aims to make a prediction lying in a discrete set.”
Generalized Energy Based Models (2020, methodology): “More recently, (Belanger and McCallum, 2016; Xie et al., 2016; 2017; 2018c; 2019; Tu and Gimpel, 2018; Du and Mordatch, 2019; Deng et al., 2020) extend the idea using more sophisticated models and MCMC sampling strategies that lead to higher quality estimators.”
Deep Structured Prediction with Nonlinear Output Transformations (2018, methodology): “Multilabel Classification: For this set of experiments, we compare against SPENs on the Bibtex and Bookmarks datasets used by Belanger and McCallum [3] and Tu and Gimpel [55].”