Sequence Level Training with Recurrent Neural Networks

By Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, Wojciech Zaremba. 2015

1,780 citations recorded by the source corpus.

Many natural language processing applications use language models to generate text. These models are typically trained to predict the next word in a sequence, given the previous words and some context such as an image. However, at test time the model is expected to generate the entire sequence from scratch. This discrepancy makes generation brittle, as errors may accumulate along the way. We address this issue by proposing a novel sequence level training algorithm that directly optimizes the metric used at test time, such as BLEU or ROUGE. On three different tasks, our approach outperforms several strong baselines for greedy generation. The method is also competitive when these baselines employ beam search, while being several times faster.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

A Deep Reinforced Model for Abstractive Summarization (2017, background): “The ﬁrst one, called exposure bias (Ranzato et al., 2015), comes from the fact that the network is fully supervised at each output token during training, always knowing the ground truth sequence up to the next token to predict, but does not have such supervision when testing, hence accumulating…”
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016, background): “…alignment model [31], or by using the attention mechanism to copy rare words [37], these approaches are both unreliable at scale, since the quality of the alignments varies across languages, and the latent alignments produced by the attention mechanism are unstable when the network is deep.”
MaskGAN: Better Text Generation via Filling in the ______ (2018, methodology): “Using a REINFORCE and cross entropy hybrid, MIXER, (Ranzato et al., 2015) directly optimized BLEU score and demonstrated improvements over baselines.”
Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2023, background): “These methods represent a convergence of two bodies of work: one body of work on training language models with reinforcement learning for a variety of objectives [33, 27, 45] and another body of work on general methods for learning from human preferences [12, 19].”
Fine-Tuning Language Models from Human Preferences (2019, methodology): “This copying behavior emerged naturally from the data collection and training process; we did not use any explicit architectural mechanism for copying as in See et al. (2017); Gehrmann et al.”
Defending Against Neural Fake News (2019, background): “Models maximizing Equation 1 are trained only conditioned on human-written text, never on its own generations, creating a problem known as exposure bias (Ranzato et al., 2016).”
Get To The Point: Summarization with Pointer-Generator Networks (2017, background): “, 2016), variational autoencoders (Miao and Blunsom, 2016), and direct optimization of the performance metric (Ranzato et al., 2016), further improving performance on those datasets.”
Learning to summarize from human feedback (2020, background): “There is also a rich literature on using RL to optimize automatic metrics for NLP tasks, such as ROUGE for summarization [50, 65, 45, 15, 19], BLEU for translation [50, 66, 1, 43], and other domains [61, 27, 26].”
ColdGANs: Taming Language GANs with Cautious Sampling Strategies (2020, methodology): “In the generative architectures, the encoder part can reach such performances, supporting the hypothesis that generation failures are mostly due to the decoding step: under MLE training regimes, the decoding suffers from exposure bias [34, 2] and lacks a sequence-level loss to optimize [27].”
Residual Energy-Based Models for Text Generation (2020, background): “Second, at training time the model is conditioned on ground truth context while at test time it is conditioned on its own generations, a discrepancy referred to as exposure bias (Ranzato et al., 2016).”
Dialog-based Language Learning (2016, methodology): “Nevertheless, reinforcement learning ideas have been used to good effect for other tasks as well, such as understanding text adventure games [15], image captioning [31], machine translation and summarization [18].”
ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis (2021, background): “Their samples can still exert artifacts resulting from a sequential modeling of components, and, since these models are always trained by “teacher-forcing”, they are susceptible to an exposure bias [3, 54, 23, 60, 40].”
Cascaded Diffusion Models for High Fidelity Image Generation (2021, background): “We empirically find that conditioning augmentation is effective because it alleviates compounding error in cascading pipelines due to train-test mismatch, sometimes referred to as exposure bias in the sequence modeling literature [1, 17].”
A Convolutional Encoder Model for Neural Machine Translation (2016, methodology): “, 2014) with a similar setting to (Ranzato et al., 2015).”
Towards Coherent and Cohesive Long-form Text Generation (2018, methodology): “There are many works on training neural language models using rewards, such as Ranzato et al. (2015) and Paulus et al. (2017).”
Aligning Language Models with Preferences through f-divergence Minimization (2023, methodology): “With respect to rewards, some approaches for Machine Translation and Summarization (Ranzato et al., 2016; Bahdanau et al., 2017) directly optimize end task rewards such as BLEU and ROUGE at training time to compensate for the mismatch between the perplexity-based training of the initial model and th”