Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks

By Samy Bengio, Oriol Vinyals, Navdeep Jaitly, Noam Shazeer. 2015

2,414 citations recorded by the source corpus.

Recurrent Neural Networks can be trained to produce sequences of tokens given some input, as exemplified by recent results in machine translation and image captioning. The current approach to training them consists of maximizing the likelihood of each token in the sequence given the current (recurrent) state and the previous token. At inference, the unknown previous token is then replaced by a token generated by the model itself. This discrepancy between training and inference can yield errors that can accumulate quickly along the generated sequence. We propose a curriculum learning strategy to gently change the training process from a fully guided scheme using the true previous token, towards a less guided scheme which mostly uses the generated token instead. Experiments on several sequence prediction tasks show that this approach yields significant improvements. Moreover, it was used succesfully in our winning entry to the MSCOCO image captioning challenge, 2015.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

MaskGAN: Better Text Generation via Filling in the ______ (2018, background): “This leads to unpredictable dynamics in the hidden state of the RNN. Methods such as Professor Forcing (Lamb et al., 2016) and Scheduled Sampling (Bengio et al., 2015) have been proposed to solve this issue.”
ColdGANs: Taming Language GANs with Cautious Sampling Strategies (2020, background): “In the generative architectures, the encoder part can reach such performances, supporting the hypothesis that generation failures are mostly due to the decoding step: under MLE training regimes, the decoding suffers from exposure bias [34, 2] and lacks a sequence-level loss to optimize [27].”
Discriminative Adversarial Search for Abstractive Summarization (2020, neutral): “Inspired by Venkatra-man et al. (2015), Bengio et al. (2015) proposed a variation of Teacher Forcing wherein the ground truth tokens are incrementally replaced by the predicted words.”
ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis (2021, background): “Furthermore, the sequential sampling strategy introduces a gap between training and inference, as training relies on so-called teacher-forcing [3] (where ground truth is provided for each step) and inference is performed on previously sampled tokens.”
Cascaded Diffusion Models for High Fidelity Image Generation (2021, background): “We empirically find that conditioning augmentation is effective because it alleviates compounding error in cascading pipelines due to train-test mismatch, sometimes referred to as exposure bias in the sequence modeling literature [1, 17].”
An Actor-Critic Algorithm for Sequence Prediction (2016, methodology): “Due to this discrepancy, it has been shown that maximum likelihood training can be suboptimal (Bengio et al., 2015; Ranzato et al., 2015).”
Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, methodology): “Algorithms such as Schedule Sampling (SS) (Bengio et al., 2015), Parallel SS (Duckworth et al.”
Survey of Hallucination in Natural Language Generation (2022, methodology): “Scheduled sampling is a classic method of mitigating overexposure bias first proposed by [10].”
Attention on Attention for Image Captioning (2019, neutral): “05 every 5 epochs [5].”
VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research (2019, methodology): “Schedule sampling [7] is employed to train the models.”
Self-Critical Sequence Training for Image Captioning (2016, background): “In [8] they show that feeding back the model’s own predictions and slowly increasing the feedback probability p during training leads to significantly better test-time performance.”
Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments (2017, methodology): “language generation [8]).”
Controlling Computation versus Quality for Neural Sequence Models (2020, background): “As a majority of conditional com- putation approaches make use of stochastic binary units that pose trainability challenges, Bengio et al. (2015a) cast the problem as a reinforcement learning problem and proposed a policy that maps the activations of layers to Bernoulli masks.”
Model-Based Reinforcement Learning for Atari (2019, methodology): “Following Bengio et al. (2015); Venkatraman et al. (2016), we mitigate this problem by randomly replacing in training some frames of the input X by the prediction from the previous step while linearly increasing the mixing probability to 100% around the middle of the ﬁrst iteration of the training…”
Learning Latent Dynamics for Planning from Pixels (2018, background): “For a model of limited capacity and restricted posterior family, the solution to this objective does not necessarily coincide with the best multi-step predictions (Bengio et al., 2015; Talvitie, 2014).”
TreeQN and ATreeC: Differentiable Tree Planning for Deep Reinforcement Learning (2017, background): “Variants of scheduled sampling (Bengio et al., 2015) may be used to improve robustness of these models, but scaling to complex domains has proven challenging (Talvitie, 2014).”
Listen, attend and spell: A neural network for large vocabulary conversational speech recognition (2015, methodology): “ssing and the predictions can suffer because the model was not trained to be resilient to feeding in bad predictions at some time steps. To ameliorate this effect, we use a trick that was proposed in [19]. During training, instead of always feeding in the ground truth transcript for next step predic”
SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient (2016, methodology): “The third one is scheduled sampling (Bengio et al. 2015).”
Achieving Human Parity on Automatic Chinese to English News Translation (2018, background): “4 Beyond the Left-to-Right Bias Current NMT systems suffer from the exposure bias problem [5].”
State-of-the-Art Speech Recognition with Sequence-to-Sequence Models (2017, background): “Second, we include scheduled sampling (SS) [13, 2], which feeds the previous label prediction during training rather than ground truth.”