Mutual Information and Diverse Decoding Improve Neural Machine Translation

By Jiwei Li, Dan Jurafsky. 2016

128 citations recorded by the source corpus.

Sequence-to-sequence neural translation models learn semantic and syntactic relations between sentence pairs by optimizing the likelihood of the target given the source, i.e., $p(y|x)$, an objective that ignores other potentially useful sources of information. We introduce an alternative objective function for neural MT that maximizes the mutual information between the source and target sentences, modeling the bi-directional dependency of sources and targets. We implement the model with a simple re-ranking method, and also introduce a decoding algorithm that increases diversity in the N-best list produced by the first pass. Applied to the WMT German/English and French/English tasks, the proposed models offers a consistent performance boost on both standard LSTM and attention-based neural MT architectures.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, background): “The reason is that beam search yields a lower diversity in the outputs (Li & Jurafsky, 2016), while in self-consistency the diversity of the reasoning paths is the key to a better performance.”
Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models (2016, methodology): “Li et al., 2015; Li & Jurafsky, 2016).”
Paragraph-level Neural Question Generation with Maxout Pointer and Gated Self-attention Networks (2018, methodology): “We would need to explicitly model diversity among candidates during generation, for example, leveraging the approach described in (Li and Jurafsky, 2016).”
Analyzing Uncertainty in Neural Machine Translation (2018, background): “…(Blunsom et al., 2008; Zhang et al., 2016) which explicitly attempt to model multiple modes in the data distribution, or decoding strategies which attempt to predict diverse outputs while leaving the model unchanged (Gimpel et al., 2013; Vijayakumar et al., 2016; Li & Jurafsky, 2016; Cho, 2016).”
Generating More Interesting Responses in Neural Conversation Models with Distributional Constraints (2018, methodology): “( Vijayakumar et al., 2016; Li and Jurafsky, 2016; Li et al., 2016b) Following previous work we evaluated our approach using a combination of automatic metrics and human judgments.”
Neural Machine Translation with Reconstruction (2016, background): “In contrast, Li and Jurafsky (2016) rescore translation candidates on sentence-level with the mutual information between source and target sides.”
Comparison of Diverse Decoding Methods from Conditional Language Models (2019, background): “(machine translation) Top-g Capping Beam Search (Li and Jurafsky, 2016) Only consider the top c hypotheses from each parent hypothesis at each time step.”
Submodular Optimization-based Diverse Paraphrasing and its Effectiveness in Data Augmentation (2019, background): “There have been some prior works (Li and Jurafsky, 2016; Elhamifar et al., 2012) which address the notion of diversity in NLP, including in sequence learning frameworks (Song et al.”
Diverse Beam Search for Improved Description of Complex Scenes (2018, methodology): “We observe that DBS outperforms BS and Li et al. (2015) while being comparable to or slightly better than Li and Jurafsky (2016) that uses an additional language model.”
On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation (2020, background): “, 2019), subword regularization (Kudo, 2018), and noisy channel modelling (Li and Jurafsky, 2016) are orthogonal to MRT.”
Decoding and Diversity in Machine Translation (2020, background): “Other studies propose decoding strategies to increase diversity, but lack comparisons to the ground truth distribution (Gimpel et al., 2013; Vijayakumar et al., 2016; Li and Jurafsky, 2016; Cho, 2016).”
Towards Decoding as Continuous Optimisation in Neural Machine Translation (2017, background): “, 2015, inter alia), and reranking (Birch, 2016; Li and Jurafsky, 2016).”
Natural Language Generation for Spoken Dialogue System using RNN Encoder-Decoder Networks (2017, background): “…applied to solve variety of tasks: conversational modeling (Vinyals and Le, 2015; Li et al., 2015, 2016), machine translation (Luong et al., 2015; Li and Jurafsky, 2016) For task-oriented dialogue systems, Wen et al. (2015a) combined a forward RNN generator, a CNN reranker, and a backward RNN…”
MetaMind Neural Machine Translation System for WMT 2016 (2016, methodology): “NMT models first achieved state-of-the-art performance on the WMT English→German news-domain task in 2015 (Luong et al., 2015) and subsequent improvements have been reported since then (Sennrich et al., 2015a; Li and Jurafsky, 2016).”
Neural Machine Translation (2016, unknown): “, 2016] source-to-target and target-to-source [Li and Jurafsky, 2016]”
Trading Off Diversity and Quality in Natural Language Generation (2020, background): “The concept of oversampling generations and then ranking them has been popular since the days of statistical machine translation (Shen et al., 2004) but has also been used more recently in other domains (Li et al., 2016; Ippolito et al., 2019b; Kriz et al., 2019).”
Do Massively Pretrained Language Models Make Better Storytellers? (2019, background): “While there are many proposed solutions to the problem (Li and Jurafsky, 2016; Vijayakumar et al., 2018; Baheti et al., 2018; Zhang et al., 2018; See et al., 2019), it has been shown that a primary cause is likelihood-maximizing decoding algorithms such as greedy decoding and beam search (Holtzman…”
Noisy Parallel Approximate Decoding for Conditional Recurrent Language Model (2016, methodology): “1 , 1 } , as suggested by the authors of [16] based on the validation set performance.”