Minimum Risk Training for Neural Machine Translation

By Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, Yang Liu. 2015

480 citations recorded by the source corpus.

We propose minimum risk training for end-to-end neural machine translation. Unlike conventional maximum likelihood estimation, minimum risk training is capable of optimizing model parameters directly with respect to arbitrary evaluation metrics, which are not necessarily differentiable. Experiments show that our approach achieves significant improvements over maximum likelihood estimation on a state-of-the-art neural machine translation system across various languages pairs. Transparent to architectures, our approach can be applied to more neural networks and potentially benefit more NLP tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016, background): “…alignment model [31], or by using the attention mechanism to copy rare words [37], these approaches are both unreliable at scale, since the quality of the alignments varies across languages, and the latent alignments produced by the attention mechanism are unstable when the network is deep.”
Sequence-Level Knowledge Distillation (2016, background): “, 2011), and minimum risk training (Och, 2003; Shen et al., 2016).”
Can Neural Machine Translation be Improved with User Feedback? (2018, background): “Integrate into minimum risk training: (Shen et al., 2016) RW-MRT(θ) = S ∑”
Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning (2018, methodology): “For k > 1 this is similar to the minimum-risk training for NMT proposed in Shen et al. (2016).”
Neural Architecture Search with Reinforcement Learning (2016, neutral): “It is therefore similar to the work on BLEU optimization in Neural Machine Translation (Shen et al., 2016; Ranzato et al., 2015).”
Adversarial Learning for Neural Dialogue Generation (2017, background): “Also related is recent work (Shen et al., 2016; Wiseman and Rush, 2016) to address the issues of exposure bias and loss-evaluation mismatch in neural translation.”
Incorporating Copying Mechanism in Sequence-to-Sequence Learning (2016, background): “Seq2Seq Learning can be expressed in a probabilistic view as maximizing the likelihood (or some other evaluation metrics (Shen et al., 2015)) of observing the output (target) sequence given an input (source) sequence.”
Dual Learning for Machine Translation (2016, background): “…3: Tokenized En → Fr BLEU on newstest2014 System BLEU RNNSearch[6] 29.97 / 33.08 RNNSearch-LV with 500 k source/target words[6] 32.68 / 34.11 MRT [13] 31.30 / 34.23 dual-NMT (Large) 32.06 / 34.83 We summarize the En → Fr BLEU scores (carried out by multi-bleu.pl ) on tokenized newstest2014 of…”
Modeling Coverage for Neural Machine Translation (2016, methodology): “More specifically, we choose to maximize the likelihood of reference sentences as most other NMT models (see, however (Shen et al., 2016)):”
RMM: Reinforced Memory Management for Class-Incremental Learning (2023, methodology): “It has been widely applied to many optimization problems, e.g., neural architecture search [54, 59] and neural machine translation [38, 46].”
Reward Augmented Maximum Likelihood for Neural Structured Prediction (2016, background): “Some work in robotics has considered exploiting supervision as a means to provide indirect sampling guidance to improve policy search methods that maximize task reward [18, 19, 26], but these approaches do not make use of maximum likelihood training.”
Findings of the 2017 Conference on Machine Translation (WMT17) (2017, methodology): “The system makes extensive use of the synthetic training data provided by Junczys-Dowmunt and Grundkiewicz (2016), as well as min-risk training for fine-tuning (Shen et al., 2016).”
Nematus: a Toolkit for Neural Machine Translation (2017, background): “Additionally, Nematus supports minimum risk training (MRT) (Shen et al., 2016) to optimize towards an arbitrary, sentence-level loss function.”
Sequence-to-Sequence Learning as Beam-Search Optimization (2016, background): “Indeed, since in supervised text-generation we typically know the gold partial sequences at each time-step, it may be unnecessary to resort to full reinforcement-learning for text-generation problems.”
Ranking Sentences for Extractive Summarization with Reinforcement Learning (2018, neutral): “Advan-tageously, we do not require an online reward estimator, we pre-compute ˆ Y , which leads to a signiﬁ-cant speedup during training compared to MIXER (Ranzato et al., 2015) and related training schemes (Shen et al., 2016).”
Ask the Right Questions: Active Question Reformulation with Reinforcement Learning (2017, background): “…et al. [2016] extend this line of work using actor-critic training for MT. RL training can also prevent exposure bias ; an inconsistency between training and inference time stemming from the fact that the model never sees its own mistakes during training [Ranzato et al., 2015, Shen et al., 2016].”
A Teacher-Student Framework for Zero-Resource Neural Machine Translation (2017, methodology): “To address this problem, it is possible to construct a sub-space by either sampling (Shen et al., 2016), generating a k -best list (Cheng et al., 2016b) or mode approximation (Kim and Rush, 2016).”
Improved Image Captioning via Policy Gradient optimization of SPIDEr (2016, background): “, [13, 30, 2, 16]), but mostly in the context of machine translation.”