An Actor-Critic Algorithm for Sequence Prediction

By Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan J. Lowe, Joelle Pineau, Aaron C. Courville, Yoshua Bengio. 2016

671 citations recorded by the source corpus.

We present an approach to training neural networks to generate sequences using actor-critic methods from reinforcement learning (RL). Current log-likelihood training methods are limited by the discrepancy between their training and testing modes, as models must generate tokens conditioned on their previous guesses rather than the ground-truth tokens. We address this problem by introducing a \textit{critic} network that is trained to predict the value of an output token, given the policy of an \textit{actor} network. This results in a training procedure that is much closer to the test phase, and allows us to directly optimize for a task-specific score such as BLEU. Crucially, since we leverage these techniques in the supervised learning setting rather than the traditional RL setting, we condition the critic network on the ground-truth output. We show that our method leads to improved performance on both a synthetic task, and for German-English machine translation. Our analysis paves the way for such methods to be applied in natural language generation tasks, such as machine translation, caption generation, and dialogue modelling.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

MaskGAN: Better Text Generation via Filling in the ______ (2018, background): “More recently, actor-critic methods in natural language were explored in Bahdanau et al. (2017) where instead of having rewards supplied by a discriminator in an adversarial setting, the rewards are task-speciﬁc scores such as BLEU.”
Training language models to follow instructions with human feedback (2022, background): “…feedback as a reward in domains such as dialogue (Jaques et al., 2019; Yi et al., 2019; Hancock et al., 2019), translation (Kreutzer et al., 2018; Bahdanau et al., 2016), semantic parsing (Lawrence and Riezler, 2018), story generation (Zhou and Xu, 2020), review generation (Cho et al., 2018),…”
Learning to summarize from human feedback (2020, background): “Human feedback has also been used as a reward to train models in other domains such as dialogue [25, 68, 21], translation [32, 1], semantic parsing [34], story generation [72], review generation [7], and evidence extraction [46].”
Recursively Summarizing Books with Human Feedback (2021, background): “…et al., 2019; Stiennon et al., 2020), dialogue (Jaques et al., 2019; Yi et al., 2019; Hancock et al., 2019), translation (Kreutzer et al., 2018; Bahdanau et al., 2016), semantic parsing (Lawrence and Riezler, 2018), story generation (Zhou and Xu, 2020), review generation (Cho et al., 2018),…”
Towards Coherent and Cohesive Long-form Text Generation (2018, methodology): “In NLP, we have observed similar practices by Ranzato et al. (2015), Bahdanau et al. (2016), and Nguyen et al. (2017).”
Can Neural Machine Translation be Improved with User Feedback? (2018, background): “Nguyen et al., 2017; Kreutzer et al., 2017; Bahdanau et al., 2017).”
Aligning Language Models with Preferences through f-divergence Minimization (2023, methodology): “With respect to rewards, some approaches for Machine Translation and Summarization (Ranzato et al., 2016; Bahdanau et al., 2017) directly optimize end task rewards such as BLEU and ROUGE at training time to compensate for the mismatch between the perplexity-based training of the initial model and th”
Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, methodology): “Bahdanau et al. (2016) proposed an actor-critic algorithm to address the variance/large action space problems when using REINFORCE for language generation; follow-up works such as KG-A2C (Ammanabrolu & Hausknecht, 2020), TrufLL (Martin et al., 2022), AE-DQN (Zahavy et al., 2018), and GALAD…”
On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting (2022, methodology): “Some approaches (for instance, in machine translation and summarization (Ranzato et al., 2016; Bahdanau et al., 2017)) directly optimize performance metrics such as BLEU and ROUGE at training time.”
Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning (2018, methodology): “A typical approach is to combine REINFORCE (Williams, 1992) with policies based on deep sequence-to-sequence learning (Bahdanau et al., 2015), for example, in machine translation (Bahdanau et al., 2017), semantic parsing (Liang et al., 2017), or summarization (Paulus et al., 2017).”
Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control (2016, methodology): “Much previous work on combining RL and MLE has used MLE training simply as a way to bootstrap the training of an RL model (Ranzato et al., 2015; Bahdanau et al., 2016; Li et al., 2016), since training with RL from scratch is difficult.”
Self-Critical Sequence Training for Image Captioning (2016, background): “Actor-critic methods [14] , which instead train a second “critic” network to provide an estimate of the value of each generated word given the policy of an actor network, have also been investigated for sequence problems recently [19].”
Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization (2018, background): “Adversarial learning [6, 26] has been shown to be successful in dialog generation, translation, image captioning and a series of natural language generation tasks [7, 12, 27, 28, 29, 30, 31, 32, 33, 34].”
Adversarial Learning for Neural Dialogue Generation (2017, methodology): “The adversarial REINFORCE algorithm consists of two components: a generative model G and a discriminative model D .”
SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient (2016, methodology): “In this paper, to address the above two issues, we follow (Bachman and Precup 2015; Bahdanau et al. 2016) and consider the sequence generation procedure as a sequential decision making process.”
Professor Forcing: A New Algorithm for Training Recurrent Networks (2016, methodology): “The scheduled sampling method Bengio et al. (2015), which is closely related to SEARN (Daumé et al., 2009) and DAGGER Ross et al. (2010), involves randomly using the network’s predictions as its inputs (as in sampling mode) with some probability that increases over the course of training. This force”
Improving Neural Language Modeling via Adversarial Training (2019, methodology): “For fair comparison, we follow the standard data pre-processing procedures described in Ranzato et al. (2016); Bahdanau et al. (2017).”
End-to-End Offline Goal-Oriented Dialog Policy Learning via Policy Gradient (2017, methodology): “[2], we use reward shaping [13] to deal with this problem.”