Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation
By Bryan Eikema, Wilker Aziz. 2020
169 citations recorded by the source corpus.
Recent studies have revealed a number of pathologies of neural machine translation (NMT) systems. Hypotheses explaining these mostly suggest there is something fundamentally wrong with NMT as a model or its training algorithm, maximum likelihood estimation (MLE). Most of this evidence was gathered using maximum a posteriori (MAP) decoding, a decision rule aimed at identifying the highest-scoring translation, i.e. the mode. We argue that the evidence corroborates the inadequacy of MAP decoding more than casts doubt on the model and its training algorithm. In this work, we show that translation distributions do reproduce various statistics of the data well, but that beam search strays from such statistics. We show that some of the known pathologies and biases of NMT are due to MAP decoding and not to NMT’s statistical assumptions nor MLE. In particular, we show that the most likely translations under the model accumulate so little probability mass that the mode can be considered essentially arbitrary. We therefore advocate for the use of decision rules that take into account the translation distribution holistically. We show that an approximation to minimum Bayes risk decoding gives competitive results confirming that NMT models do capture important aspects of translation well in expectation.
Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Machine Translation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Percy Liang
- Yejin Choi
- Jianfeng Gao
- Noah A. Smith
- Graham Neubig
- Quoc V. Le
In Information Systems
- Percy Liang
- Denny Zhou
- Luke S. Zettlemoyer
- Sida I. Wang
- Quoc V. Le
- Jason Eisner
- Ethan Perez
- Michael Lewis
In Natural Language Processing
- Yejin Choi
- Luke S. Zettlemoyer
- Mohit Bansal
- Percy Liang
- Noah A. Smith
- Graham Neubig
- Jianfeng Gao
- Hannaneh Hajishirzi
In Machine Translation
- Nitika Mathur
- Ricardo Rei
- Alon Lavie
- Markus Freitag
- Craig Alan Stewart
- George F. Foster
- André F. T. Martins
- Percy Liang
Citations in the corpus
- Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, background): “…sampling (Ackley et al., 1985; Ficler & Goldberg, 2017), top-k sampling (Fan et al., 2018; Holtzman et al., 2018; Radford et al., 2019), nucleus sampling (Holtzman et al., 2020), minimum Bayes risk decoding (Eikema & Aziz, 2020; Shi et al., 2022), and typical decoding (Meister et al., 2022).”
- RL with KL penalties is better viewed as Bayesian inference (2022, background): “…likely output) are frequently of poor quality and can be substantially improved upon with decoding procedures taking into account the entire distribution, e.g. minimum Bayes risk in translation (Eikema and Aziz, 2020) or self-consistency chain-of-thought in question answering (Wang et al., 2022).”
- Typical Decoding for Natural Language Generation (2022, background): “For example, in the domain of machine translation, the most probable string under the model is often the empty string (Stahlberg and Byrne, 2019; Eikema and Aziz, 2020).”
- Energy-Based Reranking: Improving Neural Machine Translation Using Energy-Based Models (2020, result): “In parallel to our study, Eikema and Aziz (2020) also report that the mismatch regarding MLE training of autoregressive models is attributable to the distribution of the probability mass rather than the parameter estimation, resulting in a poor MAP decoding.”
- Results of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust (2022, methodology): “For Chinese → English we used the same process but replacing the NMT model with MBART50 (many-to-one) and using only 50 samples.”
- Language Model Evaluation Beyond Perplexity (2021, methodology): “More specifically, we take this as quantitative evidence for recent qualitative observations that when left to generate lots of text, neural language models based on the transformer architecture tend to babble repetitively (Holtzman et al., 2020; Cohen and Beck, 2019; Eikema and Aziz, 2020).”
- Natural Language to Code Translation with Execution (2022, methodology): “…the maximum a posteriori (MAP) method on many tasks, including syntactic parsing (Titov and Henderson, 2006; Shi et al., 2019; Zhang et al., 2020), statistical machine translation (Kumar and Byrne, 2004; Zhang and Gildea, 2008), and neural machine translation (Eikema and Aziz, 2020, 2021).”
- MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers (2021, background): “Other related work has proposed metrics that rely on multiple samples for quality-diversity evaluation [7], and Bayesian approaches to compare the distribution of statistics in machine translation [17].”
- Contrastive Decoding: Open-ended Text Generation as Optimization (2022, background): “, 2019), but the generated text is prone to incoherence and topic drift as unlucky sampling choices compound over long sequences (Eikema and Aziz, 2020; Maynez et al., 2020).”
- Better Instruction-Following Through Minimum Bayes Risk (2024, unknown): “These benefits persist across a wide range of question categories and are also consistent across models of various sizes, demonstrating that small LLM judges can be used to improve much larger LLMs at inference time.”