Correcting Length Bias in Neural Machine Translation

By Kenton Murray, David Chiang. 2018

197 citations recorded by the source corpus.

We study two problems in neural machine translation (NMT). First, in beam search, whereas a wider beam should in principle help translation, it often hurts NMT. Second, NMT has a tendency to produce translations that are too short. Here, we argue that these problems are closely related and both rooted in label bias. We show that correcting the brevity problem almost eliminates the beam problem; we compare some commonly-used methods for doing this, finding that a simple per-word reward works well; and we introduce a simple and quick way to tune this reward using the perceptron algorithm.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

PEER: A Collaborative Language Model (2022, methodology): “To make the comparison more fair, we thus split our dataset of Wikipedia intros into 100 dev examples and 400 test examples; the dev examples are exclusively used for picking the exponential length penalty (Murray and Chiang, 2018) that maximizes the model’s average Rouge-1 score.”
Masked Language Model Scoring (2019, methodology): “As 100-best can be worse than 4-best due to the beam search curse (Yang et al., 2018; Murray and Chiang, 2018), we ﬁrst decode both beam sizes to ensure no systematic degradation in our models.”
Constrained Language Models Yield Few-Shot Semantic Parsers (2021, methodology): “duce the output t by extending the prompt p using a length-normalized variant of beam search (Murray and Chiang, 2018; Wu et al., 2016).”
On NMT Search Errors and Model Errors: Cat Got Your Tongue? (2019, background): “A similar argument was made by Murray and Chiang (2018) who pointed out the difficulty for a locally normalized model to estimate the “budget” for all remaining (longer) translations.”
End-to-End Speech Recognition: A Survey (2023, background): “Very small beam sizes also partly mask a length bias exhibited by E2E attention-based encoder-decoder models [279], [280], thus trading model errors against search errors [281].”
Neural Machine Translation: A Review (2019, methodology): “In contrast, traditional (symbolic) MT systems can easily use very large vocabularies (Heafield et al., 2013; Lin & Dyer, 2010; Chiang, 2007; Koehn, 2010).”
Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation (2020, methodology): “Later studies have also confirmed the same connection (Blain et al., 2017; Murray and Chiang, 2018; Yang et al., 2018; Kumar and Sarawagi, 2019).”
Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation (2023, methodology): “As is widely noted (Murray & Chiang, 2018) and especially in the context of NLG uncertainty by Malinin & Gales (2020), in expectation longer sequences have lower joint likelihoods because of the conditional independence of the token probabilities.”
Pre-Trained Language Models for Text Generation: A Survey (2022, background): “According to prior work [169], beam search is a decent strategy for tasks requiring accuracy, such as machine translation and text summarization.”
How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering (2020, background): “However, it is well-attested that MLE-trained language generators are biased, tending to prefer short outputs (Murray and Chiang, 2018), or being biased towards more frequent vocabulary (Ott et al.”
If Beam Search Is the Answer, What Was the Question? (2020, methodology): “Methods such as length and coverage normalization (Jean et al., 2015; Tu et al., 2016; Murray and Chiang, 2018), which augment the MAP objective with an additive term or multiplicative factor, have been adopted to alleviate these issues.”
Language Model Evaluation Beyond Perplexity (2021, unknown): “guage generation models (and by those in these experiments) has been claimed to result in models that favor shorter than typical sequences (Sountsov and Sarawagi, 2016; Murray and Chiang, 2018).”
Best-First Beam Search (2020, methodology): “…NLP, most research on beam search has focused on altering the log-probability scoring function to return improved results, e.g., higher BLEU scores (Wu et al., 2016; Murray and Chiang, 2018; Shu and Nakayama, 2018; Yang et al., 2018) or a more diverse set of outputs (Vijayakumar et al., 2016).”
Task-Oriented Dialogue as Dataflow Synthesis (2020, background): “This behavior is not specific to our system: under-length predictions are also welldocumented in neural machine translation systems (Murray and Chiang, 2018).”
On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation (2020, background): “The beam search problem has previously been linked to length bias (Yang et al., 2018; Murray and Chiang, 2018) and the copy mode (Ott et al.”
Detecting hallucinations in large language models using semantic entropy (2024, unknown): “Length normalization has some empirical success 48 , including in our own preliminary experiments, but little theoretical justification in the literature.”
The EOS Decision and Length Extrapolation (2020, background): “Extrapolation has also been a challenge for neural machine translation; Murray and Chiang (2018) identiﬁes models producing translations that are too short as one of the main challenges for neural MT.”
A Plug-and-Play Method for Controlled Text Generation (2021, unknown): “This process allows a decoding algorithm to choose among a set of candidates, potentially taking into account global qualities of a sequence, such as length (Murray and Chiang, 2018) or coverage (Wu et al.”
Empirical Analysis of Beam Search Performance Degradation in Neural Sequence Models (2019, background): “A recent line of work in machine translation suggested the performance degradation is due to length bias (Yang et al., 2018; Murray & Chiang, 2018).”
The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarks (2022, background): “, a model’s positional bias (Murray and Chiang, 2018; Ko et al., 2020) (bias to resolve “she" to a closeby entity) or spurious correlations (Schlegel et al.”