Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback

By Khanh Nguyen, Hal Daumé Iii, Jordan L. Boyd-Graber. 2017

145 citations recorded by the source corpus.

Machine translation is a natural candidate problem for reinforcement learning from human feedback: users provide quick, dirty ratings on candidate translations to guide a system to improve. Yet, current neural machine translation training focuses on expensive human-generated reference translations. We describe a reinforcement learning algorithm that improves neural machine translation systems from simulated human feedback. Our algorithm combines the advantage actor-critic algorithm (Mnih et al., 2016) with the attention-based neural encoder-decoder architecture (Luong et al., 2015). This algorithm (a) is well-designed for problems with a large action space and delayed rewards, (b) effectively optimizes traditional corpus-level machine translation metrics, and (c) is robust to skewed, high-variance, granular feedback modeled after actual human behaviors.

Subfields: Artificial Intelligence, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Fine-Tuning Language Models from Human Preferences (2019, methodology): “Nguyen et al. (2017) used RL on BLEU but applied several error models to approximate human behavior.”
Learning to summarize from human feedback (2020, background): “There is also a rich literature on using RL to optimize automatic metrics for NLP tasks, such as ROUGE for summarization [50, 65, 45, 15, 19], BLEU for translation [50, 66, 1, 43], and other domains [61, 27, 26].”
Towards Coherent and Cohesive Long-form Text Generation (2018, methodology): “In NLP, we have observed similar practices by Ranzato et al. (2015), Bahdanau et al. (2016), and Nguyen et al. (2017).”
Can Neural Machine Translation be Improved with User Feedback? (2018, methodology): “Nguyen et al., 2017; Kreutzer et al., 2017; Bahdanau et al., 2017).”
Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, methodology): “RL for NLP. RL, often in the form of bandit learning, has been used to improve models in machine translation (Wu et al., 2016; Nguyen et al., 2017; Kiegeland & Kreutzer, 2021), summarization (Stiennon et al., 2020; Paulus et al., 2017), dialogue (Li et al., 2016; Zhou et al., 2017; Jaques et al.,…”
Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning (2018, methodology): “Online actorcritic methods have been applied to sequenceto-sequence RL by Bahdanau et al. (2017) and Nguyen et al. (2017).”
Learning to Extract Coherent Summary via Deep Reinforcement Learning (2018, methodology): “It is often used in settings where the objective is not differentiable with respect to the model parameters, such as works done by (Paulus, Xiong, and Socher 2017; Nguyen, Boyd-Graber, and Daume 2017).”
Training Diffusion Models with Reinforcement Learning (2023, unknown): “A number of works have studied using human feedback to optimize models in settings such as simulated robotic control [7], game-playing [24], machine translation [32], citation retrieval [28], browsing-based question-answering [31], summarization [49, 61], instruction-following [34], and alignment wi”
A Reinforcement Learning Approach to Interactive-Predictive Neural Machine Translation (2018, methodology): “(7) in Nguyen et al. (2017)) 16: end if 17: Update γt = γt−1 + 1t ( H̄(ŷ1:t)− γt−1 ) 18: break if <eos> in ŷ1:t 19: end for 20: end for updates of the parameters of the policy (line 14), critic (line 15), and average entropy (line 17) are performed.”
Interactive Learning from Activity Description (2021, background): “, star ratings) [28, 29, 50], this setting is suitable for real-world applications.”
Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translation (2021, methodology): “It has furthermore been identi-ﬁed as a promising tool to adapt pretrained models to new domains or user preferences by replacing reward functions with human feedback in human-in-the-loop learning (Sokolov et al., 2016; Nguyen et al., 2017).”
Learning to Actively Learn Neural Machine Translation (2018, methodology): “These methods rely on deep reinforcement learning framework (Yue et al., 2012; Wirth et al., 2017) or bandit algorithms (Nguyen et al., 2017).”
Counterfactual Learning for Machine Translation: Degeneracies and Solutions (2017, background): “Recent research has proposed bandit structured prediction (Sokolov et al., 2016; Kreutzer et al., 2017; Nguyen et al., 2017) for online learning of machine translation from weak user feedback to predicted translations, instead of from costly manually created reference translations.”
AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback (2023, methodology): “RL research applied to NLP has a long history [73, 60, 27, 48, 41, 30, 29, 51, 59], and we expect future work in this direction to benefit from the ideas and artifacts in AlpacaFarm.”
Offline Reinforcement Learning from Human Feedback in Real-World Sequence-to-Sequence Tasks (2020, background): “Nguyen et al. (2017) illustrated how badly machine translation systems can handle human-level noise in direct feedback for online RL with simulations.”
CoaCor: Code Annotation for Code Retrieval with Reinforcement Learning (2019, methodology): “, the NL vocabulary) is large and suffer from biases when estimating the return of rarely taken actions [39], leading to unstable training.”