Residual Energy-Based Models for Text Generation

By Yuntian Deng, Anton Bakhtin, Myle Ott, Arthur Szlam. 2020

150 citations recorded by the source corpus.

Text generation is ubiquitous in many NLP tasks, from summarization, to dialogue and machine translation. The dominant parametric approach is based on locally normalized models which predict one word at a time. While these work remarkably well, they are plagued by exposure bias due to the greedy nature of the generation process. In this work, we investigate un-normalized energy-based models (EBMs) which operate not at the token but at the sequence level. In order to make training tractable, we first work in the residual of a pretrained locally normalized language model and second we train using noise contrastive estimation. Furthermore, since the EBM works at the sequence level, we can leverage pretrained bi-directional contextual representations, such as BERT and RoBERTa. Our experiments on two large language modeling datasets show that residual EBMs yield lower perplexity compared to locally normalized baselines. Moreover, generation via importance sampling is very efficient and of higher quality than the baseline models according to human evaluation.

Subfields: Artificial Intelligence, Natural Language Processing, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Generative Models

Citations in the corpus

Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, background): “The same re-ranking strategy for LLMs was also proposed in Deng et al. (2019), where the reward is seen as an energy function.”
Denoising Diffusion Probabilistic Models (2020, neutral): “By the known connection between score matching and energy-based modeling, our work could have implications for other recent work on energy-based models [11, 38, 15, 8].”
Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling (2020, neutral): “In [22], the authors use the idea of training an EBM deﬁned jointly by a generator and an additional critic function in the text generation setting.”
Implicit Behavioral Cloning (2021, background): “In applications, 222 energy based models have recently shown state-of-the-art results across a number of domains, including 223 various computer vision tasks [35, 36], as well as generative modeling tasks such as image and text 224 generation [12, 37, 38].”
A Distributional Approach to Controlled Text Generation (2020, background): “Some current applications to text generation include Parshakova et al. (2019a) and Deng et al. (2020), who augment a standard autoregressive LM with an additional global factor in order to get a lower perplexity on the training data.”
An approximate sampler for energy-based models with divergence diagnostics (2022, methodology): “Deng et al. (2020) train globally normalized language models to combat negative effects of local normalization, and use a form of sampling importance resampling (Rubin, 1987) to sample from the resulting EBM using an autoregressive proposal language model.”
Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generation (2022, background): “…et al., 2016; Shen et al., 2016; Bahdanau et al., 2017; Chen et al., 2020), learning to search (Leblond et al., 2018), energy-based models (Deng et al., 2020), imitation learning (Du and Ji, 2019), generative adversarial networks (Yu et al., 2017) and knowledge distillation (Liu et al., 2019).”
Exposing the Implicit Energy Networks behind Masked Language Models via Metropolis-Hastings (2021, methodology): “While we primarily focus on sampling and generation, our ﬁndings open up avenues for devising more direct, stable and simple training (Deng et al., 2020) procedures for large-scale energy-based sequence models inspired from the MLM objectives and our proposed MH sampling scheme.”
Energy-Based Reranking: Improving Neural Machine Translation Using Energy-Based Models (2020, methodology): “Then we resam-ple two sentences, y 1 and y 2 , from the renormalized set, which deﬁnes a conditional distribution: k exp( − E θ ( y k , x ) /T ) (a similar sampling approach has been used in Deng et al. (2020)).”
Refining Deep Generative Models via Discriminator Gradient Flow (2020, background): “In energy-based generative models (Arbel et al., 2021; Deng et al., 2020), the energy functions can be viewed as a component that improves some base generator.”
Pre-Training Transformers as Energy-Based Cloze Models (2020, methodology): “While EBMs have previously been applied to left-to-right (Wang et al., 2015) or globally normalized (Rosenfeld et al., 2001; Deng et al., 2020) text generation, they have not previously been applied to cloze models or for pre-training NLP models.”
Generalized Energy Based Models (2020, methodology): “More recently, (Belanger and McCallum, 2016; Xie et al., 2016; 2017; 2018c; 2019; Tu and Gimpel, 2018; Du and Mordatch, 2019; Deng et al., 2020) extend the idea using more sophisticated models and MCMC sampling strategies that lead to higher quality estimators.”
Adding A Filter Based on The Discriminator to Improve Unconditional Text Generation (2020, methodology): “[8] uses the energy-based models to improve the distribution of generated texts.”
COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics (2022, unknown): “Recent work has thus made attempts to train text-based EBMs each for specific tasks (Deng et al., 2020; Khalifa et al., 2020; Parshakova et al., 2019; Hu et al., 2018).”
Improved Contrastive Divergence Training of Energy Based Models (2020, background): “…2019; Grathwohl et al., 2019), compositional generation (Hinton, 1999; Du et al., 2020a), memory modeling (Bartunov et al., 2019), text generation (Deng et al., 2020), video generation (Xie et al., 2017), reinforcement learning (Haarnoja et al., 2017; Du et al., 2019), continual learning (Li et…”
Better Language Model with Hypernym Class Prediction (2022, background): “In parallel, alternative training strategies have also been proposed (Guu et al., 2020; Ziegler and Rush, 2019; Deng et al., 2020).”
Energy-Based Models for Code Generation under Compilability Constraints (2021, background): “Parshakova et al. (2019b) and Deng et al. (2020) augment a autoregressive language models with an additional global factor to obtain a lower perplexity on the training data.”
Unpaired Image-to-Image Translation via Latent Energy Transport (2020, background): “Energy-based Model Recently, energy-based generative models have drawn significant attention [40, 39, 9, 29, 7, 11, 43, 41].”