On NMT Search Errors and Model Errors: Cat Got Your Tongue?

By Felix Stahlberg, Bill Byrne. 2019

169 citations recorded by the source corpus.

We report on search errors and model errors in neural machine translation (NMT). We present an exact inference procedure for neural sequence models based on a combination of beam search and depth-first search. We use our exact search to find the global best model scores under a Transformer base model for the entire WMT15 English-German test set. Surprisingly, beam search fails to find these global best model scores in most cases, even with a very large beam size of 100. For more than 50% of the sentences, the model in fact assigns its global best score to the empty translation, revealing a massive failure of neural models in properly accounting for adequacy. We show by constraining search with a minimum translation length that at the root of the problem of empty translations lies an inherent bias towards shorter translations. We conclude that vanilla NMT in its current form requires just the right amount of beam search errors, which, from a modelling perspective, is a highly unsatisfactory conclusion indeed, as the model often prefers an empty translation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

The Curious Case of Neural Text Degeneration (2019, background): “Still, similar issues have been reported when using large beam sizes (Koehn & Knowles, 2017) and more recently with exact inference (Stahlberg & Byrne, 2019), a counter-intuitive observation since more comprehensive search helps maximize probability.”
AmbigQA: Answering Ambiguous Open-domain Questions (2020, background): “This problem has also been reported in other conditional generation tasks (Sountsov and Sarawagi, 2016; Stahlberg and Byrne, 2019); we leave it for future work.”
A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023, unknown): “The rationale for incorporating randomness into decoding strategies stems from the realization that high likelihood sequences often result in surprisingly low-quality text, which is called likelihood trap (Stahlberg and Byrne, 2019; Holtzman et al., 2020; Meister et al., 2020; Zhang et al., 2021).”
First the Worst: Finding Better Gender Translations During Beam Search (2021, background): “do mitigate the BLEU degradation common with larger beams (Stahlberg and Byrne, 2019).”
Controlled Text Generation as Continuous Optimization with Multiple Constraints (2021, background): “In practice, searching Y to find the highest probability generation is intractable as the space of possible sequences grows exponentially with sequence length and has also been shown to produce undesirable solutions [60].”
A Systematic Investigation of Commonsense Understanding in Large Language Models (2021, methodology): “…s () ), unless otherwise stated, we use entropy (token-level log probability) – the answer sequence log probability under the language model divided by its length: This formulation reduces the impact of sequence length i.e. , longer answers might have lower probability (Stahlberg and Byrne, 2019).”
If Beam Search Is the Answer, What Was the Question? (2020, background): “Neural probabilistic text generators are far from perfect; prior work has shown that they often generate text that is generic (Vinyals and Le, 2015; Li et al., 2016), unnatural (Holtzman et al., 2020), and sometimes even non-existent (Stahlberg and Byrne, 2019).”
Trading Off Diversity and Quality in Natural Language Generation (2020, background): “…translation and image captioning communities that after a certain point, increasing the beam size hurts BLEU scores and other measures of quality (Stahlberg & Byrne, 2019; Koehn & Knowles, 2017; Vinyals et al., 2016 We empirically quantify the relationship between sequence likelihoods and human…”
BRIO: Bringing Order to Abstractive Summarization (2022, background): “Specifically, increasing the beam width may introduce candidates with lower quality (Stahlberg and Byrne, 2019), and the generator may not be able to differentiate them from high-quality candidates.”
Typical Decoding for Natural Language Generation (2022, background): “For example, in the domain of machine translation, the most probable string under the model is often the empty string (Stahlberg and Byrne, 2019; Eikema and Aziz, 2020).”
Neural Machine Translation: A Review (2019, background): “Stahlberg and Byrne (2017) proposed to unfold the ensemble into a single network and shrink the unfolded network afterwards for efficient ensembling.”
Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation (2020, background): “, the mode of the distribution) is empty in many cases (Stahlberg and Byrne, 2019) and a general negative correlation exists between likelihood and quality beyond a certain likelihood value (Ott et al.”
Consistency of a Recurrent Language Model with Respect to Incomplete Decoding (2020, methodology): “Despite this success, MLE-trained neural sequence models have been shown to exhibit issues such as length bias (Sountsov & Sarawagi, 2016; Stahlberg & Byrne, 2019) and degenerate repetition (Holtz-man et al., 2019).”
On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation (2020, background): “the fact that translation quality does not increase consistently with beam size (Koehn and Knowles, 2017; Ott et al., 2018; Stahlberg and Byrne, 2019).”
Best-First Beam Search (2020, methodology): “Even before the findings of Stahlberg and Byrne (2019), it was well known that the best scoring hypothesis with respect to the traditional likelihood”
Decomposition Enhances Reasoning via Self-Evaluation Guided Decoding (2023, unknown): “Deterministic decoding methods such as greedy decoding and beam search (Jurafsky and Martin, 2009; Graves, 2012) often produce high-quality results but lack diversity (Stahlberg and Byrne, 2019; Meister et al., 2020).”