A Systematic Comparison of Smoothing Techniques for Sentence-Level BLEU
By Boxing Chen, Colin Cherry. 2014
257 citations recorded by the source corpus.
BLEU is the de facto standard machine translation (MT) evaluation metric. How- ever, because BLEU computes a geo- metric mean of n-gram precisions, it of- ten correlates poorly with human judg- ment on the sentence-level. There- fore, several smoothing techniques have been proposed. This paper systemati- cally compares 7 smoothing techniques for sentence-level BLEU. Three of them are first proposed in this paper, and they correlate better with human judgments on the sentence-level than other smoothing techniques. Moreover, we also compare the performance of using the 7 smoothing techniques in statistical machine transla- tion tuning.
Subfields: Artificial Intelligence, Evaluation & Benchmarks, Machine Translation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Luke S. Zettlemoyer
- Mohit Bansal
- Percy Liang
- Kyunghyun Cho
- Noah A. Smith
- Graham Neubig
- Yoshua Bengio
In Evaluation & Benchmarks
- Marco Turchi
- Matteo Negri
- Robin Jia
- Ondrej Bojar
- Jianfeng Gao
- Jordan L. Boyd-Graber
- Kyunghyun Cho
- Dallas Card
In Machine Translation
- Ondrej Bojar
- Matteo Negri
- Marco Turchi
- Jianfeng Gao
- Jindřich Libovický
- Jindřich Helcl
- Jian-Yun Nie
- Marcello Federico
Citations in the corpus
- Nematus: a Toolkit for Neural Machine Translation (2017, neutral): “Various MT metrics are supported as loss function, including smoothed sentence-level B LEU (Chen and Cherry, 2014), METEOR (Denkowski and Lavie, 2011), BEER (Stanojevic and Sima’an, 2014), and any interpolation of implemented metrics.”
- Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback (2017, methodology): “In our simulated study, we begin by modeling gold standard human ratings using add-onesmoothed sentence-level BLEU (Chen and Cherry, 2014).”
- Results of the WMT14 Metrics Shared Task (2014, background): “…and Bhattacharyya, 2014) M ETEOR Carnegie Mellon University (Denkowski and Lavie, 2014) AMBER, BLEU-NRC National Research Council of Canada (Chen and Cherry, 2014) P ARMESAN Charles University in Prague (Baranˇc´ıkov´a, 2014) T BLEU Charles University in Prague (Libovick´y and Pecina, 2014)…”
- With Little Power Comes Great Responsibility (2020, background): “(15)Note that this does not directly solve the problem of computing BLEU at the sentence level (Chen and Cherry, 2014), as it still mimicking the process of evaluating BLEU on a corpus.”
- Multi-Domain Neural Machine Translation through Unsupervised Adaptation (2017, methodology): “The scores are computed using the sentence-level BLEU (Chen and Cherry, 2014).”
- Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR (2020, unknown): “With NIST geometric sentence probability smoothing (Chen and Cherry, 2014).”
- Controlling hallucinations at word level in data-to-text generation (2021, unknown): “For the automatic analysis, we use five metrics: – BLEU [44] is a length-penalized precision score over n -grams, n ∈ 1 , 4 , optionally improved with a smoothing technique [4].”
- DialogWAE: Multimodal Response Generation with Conditional Wasserstein Auto-Encoder (2018, methodology): “We compute BLEU scores for n<4 using smoothing techniques (smoothing 7)2 [Chen and Cherry, 2014].”
- Learning to Coordinate Multiple Reinforcement Learning Agents for Diverse Query Reformulation (2018, unknown): “For pBLEU and pCos, lower values mean higher diversity.”
- PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable (2019, methodology): “In automatic evaluation, the following metrics are included: • BLEU (Chen and Cherry, 2014) measures the n-gram overlap between generated response and the target response.”
- End-to-End Non-Autoregressive Neural Machine Translation with Connectionist Temporal Classification (2018, methodology): “We also evaluated the translations using sentence-level BLEU score (Chen and Cherry, 2014) and measure the Pearson correlation with the length of the source sentence and the number of null symbols generated in the output.”
- Improving Neural Machine Translation with Conditional Sequence Generative Adversarial Nets (2017, methodology): “Compared to the baseline model, the proposed CSGAN-NMT model leads to improvement up to +1.7 BLEU points on average when trained on the small data set (see (1) and (4) in table 1).”
- Towards Cross-Lingual Generalization of Translation Gender Bias (2021, methodology): “18Smoothings were applied [5] and BLEU was scored using NLTK library [15].”
- Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space (2020, methodology): “We further report Self-Bleu (Zhu et al., 2018) to evaluate the diversity of generated sentences.”
- Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders (2017, background): “Smoothed Sentence-level BLEU (Chen and Cherry, 2014): BLEU is a popular metric that measures the geometric mean of modified n-gram precision with a length penalty (Pap-ineni et al., 2002; Li et al., 2015).”