Further Meta-Evaluation of Machine Translation

By Chris Callison-Burch, Cameron S. Fordyce, Philipp Koehn, Christof Monz, Josh Schroeder. 2008

304 citations recorded by the source corpus.

This paper analyzes the translation quality of machine translation systems for 10 language pairs translating between Czech, English, French, German, Hungarian, and Spanish. We report the translation quality of over 30 diverse translation systems based on a large-scale manual evaluation involving hundreds of hours of effort. We use the human judgments of the systems to analyze automatic evaluation metrics for translation quality, and we report the strength of the correlation with human judgments at both the system-level and at the sentence-level. We validate our manual evaluation methodology by measuring intra- and inter-annotator agreement, and collecting timing information.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

Findings of the 2016 Conference on Machine Translation (2016, background): “This conference builds on nine previous WMT workshops (Koehn and Monz, 2006; Callison-Burch et al., 2007, 2008, 2009, 2010, 2011, 2012; Bojar et al., 2013, 2014, 2015).”
Findings of the 2015 Workshop on Statistical Machine Translation (2015, neutral): “This workshop builds on eight previous WMT workshops (Koehn and Monz, 2006; Callison-Burch et al., 2007, 2008, 2009, 2010, 2011, 2012; Bojar et al., 2013, 2014).”
Findings of the 2014 Workshop on Statistical Machine Translation (2014, neutral): “This workshop builds on eight previous WMT workshops (Koehn and Monz, 2006; Callison-Burch et al., 2007, 2008, 2009, 2010, 2011, 2012; Bojar et al., 2013).”
Findings of the 2018 Conference on Machine Translation (WMT18) (2018, neutral): “This conference builds on twelve previous editions of WMT as workshops and conferences (Koehn and Monz, 2006; Callison-Burch et al., 2007, 2008, 2009, 2010, 2011, 2012; Bojar et al., 2013, 2014, 2015, 2016a, 2017).”
Findings of the 2017 Conference on Machine Translation (WMT17) (2017, methodology): “This conference builds on eleven previous editions of WMT as workshops and conference (Koehn and Monz, 2006; Callison-Burch et al., 2007, 2008, 2009, 2010, 2011, 2012; Bojar et al., 2013, 2014, 2015, 2016a).”
Findings of the 2019 Conference on Machine Translation (WMT19) (2019, neutral): “This conference builds on 13 previous editions of WMT as workshops and conferences (Koehn and Monz, 2006; Callison-Burch et al., 2007, 2008, 2009, 2010, 2011, 2012; Bojar et al., 2013, 2014, 2015, 2016, 2017, 2018).”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, methodology): “The upshot is that nlg evaluations increasingly rely on multiple methods, a trend that is equally visible in other areas of nlp , such as mt (Callison-Burch et al., 2007, 2008).”
Accurate Evaluation of Segment-level Machine Translation Metrics (2015, background): “…precisely the same quality judgment for individual translations from all assessors, and inevitably produce large numbers of conﬂicting assessments in the process, including from the same individual human judge (Callison-Burch et al., 2007; Callison-Burch et al., 2008; Callison-Burch et al., 2009).”
Findings of the 2009 Workshop on Statistical Machine Translation (2009, background): “The ULC metric had the strongest correlation with human judgments in WMT08 (Callison-Burch et al., 2008).”
BLEU Might Be Guilty but References Are Not Innocent (2020, background): “Meta-evaluation studies the correlation between human assessments and automatic evaluations (Callison-Burch et al., 2006, 2008; CallisonBurch, 2009).”
Automatic Evaluation of Linguistic Quality in Multi-Document Summarization (2010, background): “For example, at the 2008 ACL Workshop on Statistical Machine Translation, all ﬁfteen automatic evaluation metrics, including variants of BLEU scores, achieved between 42% and 56% pairwise accuracy with human judgments at the sentence level (Callison-Burch et al., 2008).”
Mining the Correlation between Human and Automatic Evaluation at Sentence Level (2010, methodology): “The correlation between automatic evaluation and human evaluation at sentence level was obtained following the practice of Callison-Burch et al. (2008).”
Findings of the 2013 Workshop on Statistical Machine Translation (2013, unknown): “This workshop builds on seven previous WMT workshops (Koehn and Monz, 2006; Callison-Burch et al., 2007, 2008, 2009, 2010, 2011, 2012).”
Findings of the 2011 Workshop on Statistical Machine Translation (2011, background): “This workshop builds on five previous WMT workshops (Koehn and Monz, 2006; Callison-Burch et al., 2007; Callison-Burch et al., 2008; Callison-Burch et al., 2009; Callison-Burch et al., 2010).”
Findings of the 2010 Joint Workshop on Statistical Machine Translation and Metrics for Machine Translation (2010, methodology): “This builds on four previous WMT workshops (Koehn and Monz, 2006; Callison-Burch et al., 2007; Callison-Burch et al., 2008; Callison-Burch et al., 2009), and one previous MetricsMATR meeting (Przybocki et al., 2008).”
Judge the Judges: A Large-Scale Evaluation Study of Neural Language Models for Online Review Generation (2019, background): “Similarly, BLEU and METEOR present highest correlations with human judgements (Callison-Burch et al., 2008), (Graham”
Findings of the 2012 Workshop on Statistical Machine Translation (2012, methodology): “In the following years (Callison-Burch et al., 2007; Callison-Burch et al., 2008; Callison-Burch et al., 2009; Callison-Burch et al., 2010; Callison-Burch et al., 2011), we abandoned the idea of using fluency and adequacy judgments, since they showed to be less reliable than simple ranking of system”
Experiments in Morphosyntactic Processing for Translating to and from German (2009, result): “For translation into English METEOR had superior correlation with human rankings to BLEU at WMT 2008 (Callison-Burch et al., 2008).”
Re-evaluating Automatic Summarization with BLEU and 192 Shades of ROUGE (2015, methodology): “In MT evaluation of metrics, although experimentation has taken place with regards to methods of elicitation of assessments from human judges (Callison-Burch et al., 2008), human assessment is always aimed to encapsulate the over-”