On the Weaknesses of Reinforcement Learning for Neural Machine Translation

By Leshem Choshen, Lior Fox, Zohar Aizenbud, Omri Abend. 2019

127 citations recorded by the source corpus.

Reinforcement learning (RL) is frequently used to increase performance in text generation tasks, including machine translation (MT), notably through the use of Minimum Risk Training (MRT) and Generative Adversarial Networks (GAN). However, little is known about what and how these methods learn in the context of MT. We prove that one of the most common RL methods for MT does not optimize the expected reward, as well as show that other methods take an infeasibly long time to converge. In fact, our results suggest that RL practices in MT are likely to improve performance only where the pre-trained parameters are already close to yielding the correct translation. Our findings further suggest that observed gains may be due to effects unrelated to the training signal, but rather from changes in the shape of the distribution curve.

Subfields: Artificial Intelligence, Natural Language Processing, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Machine Translation

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “Choshen et al. (2020) show that improvements from reinforcement learning objectives in machine translation are unrelated to the training signals, but rather a side effect from changes in the model distribution curve.”
Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, background): “, 2022) but progress in this line of work is heavily hindered by a lack of open-source benchmarks and algorithmic implementations—resulting in perception that RL is a challenging paradigm for NLP (Choshen et al., 2020; Kreutzer et al., 2021).”
On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting (2022, background): “This has sparked a wide discussion on the overall effectiveness of RM for some tasks such as machine translation (Choshen et al., 2020; Kiegeland & Kreutzer, 2021).”
Boosting Search Engines with Interactive Agents (2021, neutral): “This type of modeling is closely related to the use of RL for neural machine translation, whose robustness is currently debated (Choshen et al., 2020; Kiegeland & Kreutzer, 2021).”
RL with KL penalties is better viewed as Bayesian inference (2022, background): “Degeneration of this kind was observed in multiple language generation tasks ranging from translation (Choshen et al., 2019), summarisation (Paulus et al., 2018), story generation (Tambwekar et al., 2019), video captioning (Pa-sunuru and Bansal, 2017), dialogue (Jaques et al., 2019), to code…”
Energy-Based Reranking: Improving Neural Machine Translation Using Energy-Based Models (2020, methodology): “However, for NMT, these training algorithms are often used in conjunction with MLE training (Wu et al., 2018) or as ﬁne-tuning (Choshen et al., 2020).”
Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (2023, methodology): “As we can see, DPO, RSO, and GSHF significantly outperform the SFT baseline, and the GSHF algorithms further outperform the stronger baselines including both DPO and RSO in terms of gold reward, and GPT4 evaluations.”
Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translation (2021, methodology): “1 (Ranzato et al., 2016; Kreutzer et al., 2017; Choshen et al., 2020), which results in the following update rule with learning rate α : The temperature τ of the softmax distribution exp( y i /τ ) / j exp( y j /τ ) can be used to control the amount of exploration during learning.”
Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts (2024, background): “However, getting the PPO work is challenging in the context of LLMs [Choshen et al., 2019; Engstrom et al., 2020].”
Text Generation with Efficient (Soft) Q-Learning (2021, methodology): “Previous literature either initializes the policy π θ with MLE training, and/or use a combination of MLE and PG updates, which often leads to marginal gains in practice [63, 7].”
Text Generation by Learning from Demonstrations (2020, unknown): “However, optimizing this objective is notoriously difﬁcult, and it remains unclear if RL is beneﬁcial to text generation (Wu et al., 2018; Choshen et al., 2020).”
Reward Optimization for Neural Machine Translation with Learned Metrics (2021, background): “Multiple recent studies (Wu et al., 2016; Choshen et al., 2020) have indicated that BLEU as a training signal does not contribute to the quality improvement of translations.”
LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Models (2023, background): “However, it relies on a trial-and-error approach through interaction with the environment, making it less stable and efficient than supervised learning [8].”
Learning Natural Language Generation with Truncated Reinforcement Learning (2022, background): “We thus also measure the probability mass of the ten most frequent words (Choshen et al., 2020), and refer to it as peakiness (peak).”
Offline Reinforcement Learning from Human Feedback in Real-World Sequence-to-Sequence Tasks (2020, background): “one option is to pre-train the policy on some available supervised data, which will allow the model to concentrate on reasonable areas in the output space (Choshen et al., 2020).”
RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment (2023, background): “Although PPO is a well-established DRL method with numerous studies showcasing its effectiveness [34, 11], it learns in a trial-and-error fashion by interacting with the environment and is generally significantly less stable and less efficient as compared to supervised learning [7].”
RLHF Workflow: From Reward Modeling to Online RLHF (2024, methodology): “However, it is known that even in the best case, tuning the DRL method to its best performance requires extensive efforts in hyper-parameter selection and code-level optimization (Choshen et al., 2019; Engstrom et al., 2020).”