(Meta-) Evaluation of Machine Translation

By Chris Callison-Burch, Cameron S. Fordyce, Philipp Koehn, Christof Monz, Josh Schroeder. 2007

484 citations recorded by the source corpus.

This paper evaluates the translation quality of machine translation systems for 8 language pairs: translating French, German, Spanish, and Czech to English and back. We carried out an extensive human evaluation which allowed us not only to rank the different MT systems, but also to perform higher-level analysis of the evaluation process. We measured timing and intra- and inter-annotator agreement for three types of subjective evaluation. We measured the correlation of automatic evaluation metrics with human judgments. This meta-evaluation reveals surprising facts about the most commonly used methodologies.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Findings of the 2016 Conference on Machine Translation (2016, background): “This conference builds on nine previous WMT workshops (Koehn and Monz, 2006; Callison-Burch et al., 2007, 2008, 2009, 2010, 2011, 2012; Bojar et al., 2013, 2014, 2015).”
Findings of the 2015 Workshop on Statistical Machine Translation (2015, neutral): “This workshop builds on eight previous WMT workshops (Koehn and Monz, 2006; Callison-Burch et al., 2007, 2008, 2009, 2010, 2011, 2012; Bojar et al., 2013, 2014).”
Findings of the 2014 Workshop on Statistical Machine Translation (2014, neutral): “This workshop builds on eight previous WMT workshops (Koehn and Monz, 2006; Callison-Burch et al., 2007, 2008, 2009, 2010, 2011, 2012; Bojar et al., 2013).”
Handling Divergent Reference Texts when Evaluating Table-to-Text Generation (2019, background): “rior to rating scales for comparing generated texts (Callison-Burch et al., 2007).”
Findings of the 2018 Conference on Machine Translation (WMT18) (2018, neutral): “This conference builds on twelve previous editions of WMT as workshops and conferences (Koehn and Monz, 2006; Callison-Burch et al., 2007, 2008, 2009, 2010, 2011, 2012; Bojar et al., 2013, 2014, 2015, 2016a, 2017).”
Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge (2019, background): “This setup allows to minimise the correlation between the ratings of naturalness and quality (Novikova, Dušek, and Rieser 2018; Callison-Burch et al. 2007).”
Neural Text Summarization: A Critical Evaluation (2019, positive): “Similar lines of research, where the authors put under scrutiny existing methodologies, datasets, or models were conducted by Callison-Burch et al. (2006, 2007); Tan et al. (2015); Post (2018) in machine translation, (Gkatzia and Mahamood, 2015; Reiter and Belz, 2009; Reiter, 2018) in natu-ral…”
Findings of the 2017 Conference on Machine Translation (WMT17) (2017, methodology): “This conference builds on eleven previous editions of WMT as workshops and conference (Koehn and Monz, 2006; Callison-Burch et al., 2007, 2008, 2009, 2010, 2011, 2012; Bojar et al., 2013, 2014, 2015, 2016a).”
Findings of the 2019 Conference on Machine Translation (WMT19) (2019, neutral): “This conference builds on 13 previous editions of WMT as workshops and conferences (Koehn and Monz, 2006; Callison-Burch et al., 2007, 2008, 2009, 2010, 2011, 2012; Bojar et al., 2013, 2014, 2015, 2016, 2017, 2018).”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, methodology): “The upshot is that nlg evaluations increasingly rely on multiple methods, a trend that is equally visible in other areas of nlp , such as mt (Callison-Burch et al., 2007, 2008).”
Accurate Evaluation of Segment-level Machine Translation Metrics (2015, background): “the process, including from the same individual human judge (Callison-Burch et al., 2007; CallisonBurch et al., 2008; Callison-Burch et al., 2009).”
Findings of the 2009 Workshop on Statistical Machine Translation (2009, background): “This paper presents the results of the shared tasks of the 2009 EACL Workshop on Statistical Machine Translation, which builds on three previous workshops (Koehn and Monz, 2006; Callison-Burch et al., 2007; Callison-Burch et al., 2008).”
Evaluation of Text Generation: A Survey (2020, background): “This comparative approach has been found to produce higher inter-annotator agreement (Callison-Burch et al., 2007) in some cases.”
Evaluating the morphological competence of Machine Translation Systems (2017, background): “It is nowadays unanimously recognized that Machine Translation (MT) engines based on the neural encoder-decoder architecture with attention (Cho et al., 2014; Bahdanau et al., 2014) constitute the new state-of-the-art in statistical MT, at least for open-domain tasks (Sennrich et al., 2016a).”
Further Meta-Evaluation of Machine Translation (2008, background): “This paper presents the results the shared tasks of the 2008 ACL Workshop on Statistical Machine Translation, which builds on two past workshops (Koehn and Monz, 2006; Callison-Burch et al., 2007).”
GO FIGURE: A Meta Evaluation of Factuality in Summarization (2020, background): “Prior work concerning evaluation of automatic metrics and human evaluation for NLG systems has mainly focused on general analysis of output quality or coherence and fluency (Callison-Burch et al., 2007; Graham, 2015; Fabbri et al., 2021), rather than factuality.”