Reward Augmented Maximum Likelihood for Neural Structured Prediction

By Mohammad Norouzi, Samy Bengio, Z. Chen, Navdeep Jaitly, Mike Schuster, Yonghui Wu, Dale Schuurmans. 2016

269 citations recorded by the source corpus.

A key problem in structured output prediction is direct optimization of the task reward function that matters for test evaluation. This paper presents a simple and computationally efficient approach to incorporate task reward into a maximum likelihood framework. By establishing a link between the log-likelihood and expected reward objectives, we show that an optimal regularized expected reward is achieved when the conditional distribution of the outputs given the inputs is proportional to their exponentiated scaled rewards. Accordingly, we present a framework to smooth the predictive probability of the outputs using their corresponding rewards. We optimize the conditional log-probability of augmented outputs that are sampled proportionally to their exponentiated scaled rewards. Experiments on neural sequence to sequence models for speech recognition and machine translation show notable improvements over a maximum likelihood baseline by using reward augmented maximum likelihood (RAML), where the rewards are defined as the negative edit distance between the outputs and the ground truth labels.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

A Deep Reinforced Model for Abstractive Summarization (2017, methodology): “This phenomenon has been observed with similar sequence generation tasks like image captioning with CIDEr (Ren-nie et al., 2016) and machine translation with BLEU (Wu et al., 2016; Norouzi et al., 2016).”
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016, background): “Several recent papers [33, 38, 31] have considered different ways of incorporating the task reward into optimization of neural sequence-to-sequence models.”
Aligning Language Models with Preferences through f-divergence Minimization (2023, background): “Other studies have also explored the use of forward KL divergence in RL (Peters & Schaal, 2007; Norouzi et al., 2016).”
Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control (2016, methodology): “The RAML approach, while an important contribution, is only viable if it is possible to generate additional MLE training samples that are similar in terms of the reward function to the ground truth (i.e. samples within a small edit distance).”
Trust-PCL: An Off-Policy Trust Region Method for Continuous Control (2017, background): “Norouzi et al. (2016) make the observation that entropy regularized expected reward may be expressed as a reversed KL divergence D KL ( θ, θ ∗ ) between the policy and optimal parameters, which suggests that an alternative trust region should be used whenever such regularization is present:…”
Bridging the Gap Between Value and Policy Based Reinforcement Learning (2017, background): “Recent work has noted some connections between value and policy based RL (Norouzi et al., 2016; O’Donoghue et al., 2017; Nachum et al., 2017), when an entropy regularized objective is considered.”
PGQ: Combining policy gradient and Q-learning (2016, background): “In Norouzi et al. (2016) the authors augment a maximum likelihood objec-tive with a reward in a supervised learning setting, and develop a connection that resembles the one we develop here between the policy and the Q-values.”
Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision (2016, methodology): “Therefore, we base our training procedure on REINFORCE (Williams, 1992; Norouzi et al., 2016).”
Regularizing Neural Networks by Penalizing Confident Output Distributions (2017, background): “For more details, see Norouzi et al. (2016).”
A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models (2016, methodology): “Several recent works have proposed methods for merging maximum likelihood objectives and known reward functions for training sequential language generation models and rely on surrogate reward function such as BLEU score or edit distance [20, 16, 2].”
Insertion Transformer: Flexible Sequence Generation via Insertion Operations (2019, methodology): “…corresponding to location l : We use the negative distance function − d l as the reward function for a softmax weighting policy w l (Rusu et al., 2016; Norouzi et al., 2016) (see Figure 2 for an illustration): Next we deﬁne the slot loss at location l as a weighted A visualization of the…”
SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translation (2018, background): “…et al., 2016a; Gal and Ghahramani, 2016), which uniformly set some word embeddings to 0 at random, and Reward Augmented Maximum Likelihood (RAML; Norouzi et al. (2016)), whose implementation essentially replaces some words in the target sentences with other words from the target vocabulary.”
Improved Image Captioning via Policy Gradient optimization of SPIDEr (2016, background): “, [13, 30, 2, 16]), but mostly in the context of machine translation.”
Optimal Completion Distillation for Sequence Learning (2018, background): “On the other hand, despite many attempts to mitigate the limitations of Maximum Likelihood Estimation (MLE) (Ranzato et al., 2016; Wiseman and Rush, 2016; Norouzi et al., 2016; Bahdanau et al., 2017; Leblond et al., 2018), MLE is still considered the dominant approach for training seq2seq models.”
Quark: Controllable Text Generation with Reinforced Unlearning (2022, background): “Recent works have explored applying RL on tasks such as question-answering [84, 85, 47, 83, 84], summarization [58, 53, 70, 60, 17, 51], and machine translation [58, 87, 79, 82, 81, 13, 66, 5, 49].”
Discrete Sequential Prediction of Continuous Actions for Deep RL (2017, methodology): “In the context of reinforcement learning, sequential or autoregressive policies have previously been used to describe exponentially large action spaces such as the space of neural architectures, [50] and over sequences of words [30, 38].”
From Language to Programs: Bridging Reinforcement Learning and Maximum Marginal Likelihood (2017, methodology): “It is tempting to group our approach with sequence learning methods which interpolate between supervised learning and reinforcement learning (Ranzato et al., 2015; Venkatraman et al., 2015; Ross et al., 2011; Norouzi et al., 2016; Bengio et al., 2015; Levine, 2014).”
Reinforced Mnemonic Reader for Machine Comprehension (2017, methodology): “Besides, most of previous models use the boundary detecting method proposed by (Wang and Jiang 2017) to train their model, which is equivalent to optimizing the Exact Match (EM) metric (Norouzi et al. 2016).”