A Study of Reinforcement Learning for Neural Machine Translation

By Lijun Wu, Fei Tian, Tao Qin, J. Lai, Tie-Yan Liu. 2018

189 citations recorded by the source corpus.

Recent studies have shown that reinforcement learning (RL) is an effective approach for improving the performance of neural machine translation (NMT) system. However, due to its instability, successfully RL training is challenging, especially in real-world systems where deep models and large datasets are leveraged. In this paper, taking several large-scale translation tasks as testbeds, we conduct a systematic study on how to train better NMT models using reinforcement learning. We provide a comprehensive comparison of several important factors (e.g., baseline reward, reward shaping) in RL training. Furthermore, to fill in the gap that it remains unclear whether RL is still beneficial when monolingual data is used, we propose a new method to leverage RL to further boost the performance of NMT systems trained with source/target monolingual data. By integrating all our findings, we obtain competitive results on WMT14 English-German, WMT17 English-Chinese, and WMT17 Chinese-English translation tasks, especially setting a state-of-the-art performance on WMT17 Chinese-English translation task.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Microsoft Research Asia’s Systems for WMT19 (2019, methodology): “Furthermore, some other techniques such as de-liberation learning (Xia et al., 2017b), adversarial learning (Wu et al., 2018b), and reinforcement learning (He et al., 2017; Wu et al., 2018a) could also hep and are worthy of exploration.”
RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (2023, methodology): “While initial applications of RL on tasks such as translation (Wu et al., 2016; 2018) and summarization (Gao et al., 2019; Wu & Hu, 2018) used automatic evaluation metrics as rewards, such simpliﬁed formulations of rewards did not fully align with human notions of quality.”
An Overview of Natural Language State Representation for Reinforcement Learning (2020, methodology): “, 2017) were also used in machine translation (Wu et al., 2018).”
Machine Translation With Weakly Paired Bilingual Documents (2018, methodology): “Using monolingual data to boost the machine translation performance has attracted a lot of attention in the literature (Gulcehre et al., 2015; Sennrich et al., 2016a; Zhang & Zong, 2016; Wu et al., 2018; He et al., 2016), especially when the bilingual supervision is limited.”
Look Harder: A Neural Machine Translation Model with Hard Attention (2019, methodology): “To compute the hard-attention between the last layers of the Transformer encoder-decoder blocks, we replace the second sub-layer of the de-coder block’s last layer with the RL agent based attention mechanism.”
Exploiting Monolingual Data at Scale for Neural Machine Translation (2019, background): “…data is an important research topic for NMT and there are plenty of works studying this problem (Gulcehre et al., 2015, 2017; Sennrich et al., 2016a; He et al., 2016; Zhang and Zong, 2016; Zhang et al., 2018; Cheng et al., 2016; Wang et al., 2018; Chinea-Rios et al., 2017; Wu et al., 2018).”
Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translation (2021, result): “Learned Baseline A reward baseline can also be learned by formulating it as a regression problem, but like Wu et al. (2018) we found no empirical gains, thus excluded it from the experiments re-ported in this paper.”
Generating Persona Consistent Dialogues by Exploiting Natural Language Inference (2019, background): “Reinforcement Learning In recent years, deep reinforcement learning has been widely applied in natural language processing, such as machine translation (Wu et al. 2018), visual question generation (Fan et al.”
Energy-Based Reranking: Improving Neural Machine Translation Using Energy-Based Models (2020, methodology): “However, for NMT, these training algorithms are often used in conjunction with MLE training (Wu et al., 2018) or as ﬁne-tuning (Choshen et al., 2020).”
Modeling Fluency and Faithfulness for Diverse Neural Machine Translation (2019, methodology): “This series of work (Wu et al. 2018; Yang et al. 2018; Geng et al. 2018; Kreutzer, Uyheng, and Riezler 2018) samples a translation from all the possible translation and performs gradient descent through the translation with a sequence-level reward under the framework of reinforcement learning .”
Differentiable Sampling with Flexible Reference Word Order for Neural Machine Translation (2019, background): “Due to the high variance of the gradient estimation, training with RL can be slow and unstable (Henderson et al., 2018; Wu et al., 2018).”
On the Weaknesses of Reinforcement Learning for Neural Machine Translation (2019, methodology): “The per-token sampling as done in our experiments is more exploratory than beam search (Wu et al., 2018), reducing P K E.”
Text Generation with Efficient (Soft) Q-Learning (2021, methodology): “Previous literature either initializes the policy π θ with MLE training, and/or use a combination of MLE and PG updates, which often leads to marginal gains in practice [63, 7].”
Text Generation by Learning from Demonstrations (2020, unknown): “However, optimizing this objective is notoriously difﬁcult, and it remains unclear if RL is beneﬁcial to text generation (Wu et al., 2018; Choshen et al., 2020).”
Neural Machine Translation for Low-resource Languages: A Survey (2021, background): “[197] also argued that the above RL-based technique does not properly exploit the monolingual data and suggested several improvements.”
Neural Machine Translation: A Review (2019, background): “, 2019), sentence embedding (Lin et al., 2017; Wu et al., 2018; Zhang et al., 2018a), and reading comprehension (Hu et al.”
Learning to Teach with Dynamic Loss Functions (2018, methodology): “We choose a widely used benchmark dataset in NMT literature [42, 54, 53], released in IWSLT-14 German-English evaluation campaign [9], as the test-bed for different loss functions.”
Survey on reinforcement learning for language processing (2021, background): “However, the successful use of RL is still very challenging, especially in real-world systems using deep neural networks and huge datasets (Wu et al. 2018).”