Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics

By Chin-Yew Lin, Franz Joseph Och. 2004

927 citations recorded by the source corpus.

In this paper we describe two new objective automatic evaluation methods for machine translation. The first method is based on longest common subsequence between a candidate translation and a set of reference translations. Longest common subsequence takes into account sentence level structure similarity naturally and identifies longest co-occurring in-sequence n-grams automatically. The second method relaxes strict n-gram matching to skip-bigram matching. Skip-bigram is any pair of words in their sentence order. Skip-bigram cooccurrence statistics measure the overlap of skip-bigrams between a candidate translation and a set of reference translations. The empirical results show that both methods correlate with human judgments very well in both adequacy and fluency.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

A Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories (2016, methodology): “We compute Smoothed-BLEU (Lin and Och, 2004) score for measuring up to four-gram overlap of an alternative and the context.”
Learning to summarize from human feedback (2020, background): “Indeed, existing automatic metrics for evaluating summary quality, such as ROUGE [39], have received criticism for poor correlation with human judgments [55, 45, 6, 33].”
Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning (2018, methodology): “5), we report the numerical values returned by the COCO on-line evaluation server ‡ , using the CIDEr (Vedantam et al., 2015), ROUGE-L (Lin and Och, 2004), and METEOR (Banerjee and Lavie, 2005) metrics.”
Recursively Summarizing Books with Human Feedback (2021, methodology): “SummaQA requires paragraph-aligned summaries, which we do not have, and so we report results on ROUGE and BERTScore.”
Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning (2020, methodology): “, 2002) , ROUGE (Lin, 2004), NIST (Lin and Och, 2004), METEOR (Denkowski and Lavie, 2014) and CiDER (Vedantam et al.”
ClipCap: CLIP Prefix for Image Captioning (2021, methodology): “For the Conceptual Captions, we report the ROUGE-L [21], CIDEr, and SPICE, as suggested by the authors [33].”
UniViLM: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation (2020, methodology): “, 2002), METEOR (Banerjee and Lavie, 2005), ROUGE-L (Lin and Och, 2004) and CIDEr (Vedantam et al.”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, neutral): “B: BLEU [84]; M: METEOR [18]; R: ROUGE-L [68]; C: CIDEr-D [101].”
From captions to visual concepts and back (2014, methodology): “temmed tokens. We additionally report performance on the metrics made available from the MSCOCO captioning challenge,5 which includes scores for BLEU-1 through BLEU-4, METEOR, CIDEr [44], and ROUGE-L [27]. All of these automatic metrics are known to only roughly correlate with human judgment [10]. W”
Explain Images with Multimodal Recurrent Neural Networks (2014, background): “It also outperforms the state-of-the-art methods in terms of perplexity, B-1, B-3, and a comparable result for B-2 1.”
Comparing Automatic Evaluation Measures for Image Description (2014, background): “ROUGE measures the longest common subsequence of tokens between a candidate Y and reference X (Lin and Och, 2004).”
Image Description using Visual Dependency Representations (2013, methodology): “The model-generated descriptions are compared against the human-written gold-standard descriptions using the smoothed BLEU measure (Lin and Och, 2004).”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, background): “Thus, generation is conceived as a two-step (planning followed by realisation) process. The reliance on standard language models has one potential drawback, in that such models are founded on a local history assumption, limiting the extent to which prior selections can influence current choices. An ”
Classical Structured Prediction Losses for Sequence to Sequence Learning (2017, background): “…either BLEU or ROUGE with respect to t , the gold reference: As is common practice when computing BLEU at the sentence-level, we smooth all initial counts to one (except for unigram counts) so that the geometric mean is not dominated by zero-valued n - gram match counts (Lin and Och, 2004).”
ORANGE: a Method for Evaluating Automatic Evaluation Metrics for Machine Translation (2004, background): “S are described in details in Lin and Och (2004).”
Findings of the 2009 Workshop on Statistical Machine Translation (2009, background): “Leusch and Ney (2008) also submitted two contrastive metrics: bleusp4114, a modified version of BLEU-S (Lin and Och, 2004), with tuned n-gram weights, and bleusp, with constant weights.”
Paraphrasing Revisited with Neural Machine Translation (2017, methodology): “Sentence level BLEU is calculated using plus one smoothing (Lin and Och, 2004).”
Evaluation of Text Generation: A Survey (2020, methodology): “Although mainly designed for evaluating single- or multi-document summarization, it has also been used for evaluating short text generation, such as machine translation (Lin & Och, 2004), image captioning (Cui et al., 2018), and question generation (Nema & Khapra, 2018; Dong et al., 2019).”