Findings of the 2009 Workshop on Statistical Machine Translation

By Chris Callison-Burch, Philipp Koehn, Christof Monz, Josh Schroeder. 2009

287 citations recorded by the source corpus.

This paper presents the results of the WMT09 shared tasks, which included a translation task, a system combination task, and an evaluation task. We conducted a large-scale manual evaluation of 87 machine translation systems and 22 system combination entries. We used the ranking of these systems to measure how strongly automatic metrics correlate with human judgments of translation quality, for more than 20 metrics. We present a new evaluation technique whereby system output is edited and judged for correctness.

Subfields: Artificial Intelligence, Information Systems, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Evaluation & Benchmarks

Citations in the corpus

Findings of the 2016 Conference on Machine Translation (2016, background): “This conference builds on nine previous WMT workshops (Koehn and Monz, 2006; Callison-Burch et al., 2007, 2008, 2009, 2010, 2011, 2012; Bojar et al., 2013, 2014, 2015).”
Findings of the 2015 Workshop on Statistical Machine Translation (2015, neutral): “This workshop builds on eight previous WMT workshops (Koehn and Monz, 2006; Callison-Burch et al., 2007, 2008, 2009, 2010, 2011, 2012; Bojar et al., 2013, 2014).”
Findings of the 2014 Workshop on Statistical Machine Translation (2014, neutral): “This workshop builds on eight previous WMT workshops (Koehn and Monz, 2006; Callison-Burch et al., 2007, 2008, 2009, 2010, 2011, 2012; Bojar et al., 2013).”
Neural Approaches to Conversational AI (2018, neutral): “n judgments. However, their correlation analysis was performed at the sentence level, but decent sentence-level correlation has long been known to be difﬁcult to achieve even for machine translation (Callison-Burch et al., 2009; Graham et al., 2015), the task for which the underlying metrics (e.g., ”
Findings of the 2018 Conference on Machine Translation (WMT18) (2018, neutral): “This conference builds on twelve previous editions of WMT as workshops and conferences (Koehn and Monz, 2006; Callison-Burch et al., 2007, 2008, 2009, 2010, 2011, 2012; Bojar et al., 2013, 2014, 2015, 2016a, 2017).”
Findings of the 2017 Conference on Machine Translation (WMT17) (2017, methodology): “This conference builds on eleven previous editions of WMT as workshops and conference (Koehn and Monz, 2006; Callison-Burch et al., 2007, 2008, 2009, 2010, 2011, 2012; Bojar et al., 2013, 2014, 2015, 2016a).”
Findings of the 2019 Conference on Machine Translation (WMT19) (2019, neutral): “This conference builds on 13 previous editions of WMT as workshops and conferences (Koehn and Monz, 2006; Callison-Burch et al., 2007, 2008, 2009, 2010, 2011, 2012; Bojar et al., 2013, 2014, 2015, 2016, 2017, 2018).”
Accurate Evaluation of Segment-level Machine Translation Metrics (2015, background): “the process, including from the same individual human judge (Callison-Burch et al., 2007; CallisonBurch et al., 2008; Callison-Burch et al., 2009).”
Statistical Post-Editing for a Statistical MT System (2011, methodology): “We ran the same set of experiments for the other translation direction, French to English, which is often considered the easier of the two translation directions (Callison-Burch et al., 2009).”
Book Reviews: Data-Intensive Text Processing with MapReduce by Jimmy Lin and Chris Dyer (2010, unknown): “publicly [33].”
Data-Driven Response Generation in Social Media (2011, result): “of the WMT09 shared tasks (Callison-Burch et al., 2009).”
Asiya: An Open Toolkit for Automatic Machine Translation (Meta-)Evaluation (2010, methodology): “Specifically, we have used the French-English (fr-en) translation task from the 2009 ACL Workshop on Machine Translation, WMT09, (Callison-Burch et al., 2009).”
The Meteor metric for automatic evaluation of machine translation (2009, methodology): “Table 6 reports the correlation of several metrics with these judgments, using Spearman’s rank correlation coefficient ρ (Callison-Burch et al. 2009).”
Fast, Cheap, and Creative: Evaluating Translation Quality Using Amazon’s Mechanical Turk (2009, unknown): “The scores are more interpretable than Blue scores and than the WMT09 relative rankings, since it gives an indication of how un-derstandable the MT output is.”
Fathom: reference workloads for modern deep learning methods (2016, unknown): “seq2seq 2014 [43] Recurrent 7 Supervised WMT-15 [7] Direct language-to-language sentence translation.”
PPDB: The Paraphrase Database (2013, unknown): “…bitext corpora to extract PPDB:Eng: Europarl v7 (Koehn, 2005), consisting of bitexts for the 19 European languages, the 10 9 French-English corpus (Callison-Burch et al., 2009), the Czech, German, Spanish and French portions of the News Commentary data (Koehn and Schroeder, 2007), the United…”
Can machine translation systems be evaluated by the crowd alone (2015, background): “…despite the primary focus of the workshop being to provide a regular cross-system comparison over standardized data sets for a variety of language pairs by means of a shared translation task (Koehn and Monz, 2006; Callison-Burch et al., 2007, 2008, 2009, 2010, 2011, 2012; Bojar et al., 2013, 2014).”