Manual and Automatic Evaluation of Machine Translation between European Languages

By Philipp Koehn, Christof Monz. 2006

335 citations recorded by the source corpus.

We evaluated machine translation performance for six European language pairs that participated in a shared task: translating French, German, Spanish texts to English and back. Evaluation was done automatically using the Bleu score and manually on fluency and adequacy.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Findings of the 2016 Conference on Machine Translation (2016, background): “This conference builds on nine previous WMT workshops (Koehn and Monz, 2006; Callison-Burch et al., 2007, 2008, 2009, 2010, 2011, 2012; Bojar et al., 2013, 2014, 2015).”
Findings of the 2015 Workshop on Statistical Machine Translation (2015, background): “This workshop builds on eight previous WMT workshops (Koehn and Monz, 2006; Callison-Burch et al., 2007, 2008, 2009, 2010, 2011, 2012; Bojar et al., 2013, 2014).”
Findings of the 2014 Workshop on Statistical Machine Translation (2014, neutral): “This workshop builds on eight previous WMT workshops (Koehn and Monz, 2006; Callison-Burch et al., 2007, 2008, 2009, 2010, 2011, 2012; Bojar et al., 2013).”
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “This has been the approach in MT for over 15 years (Koehn and Monz, 2006), with metrics shared tasks organized as part of the yearly WMT workshop/conference.”
The WebNLG Challenge: Generating Text from RDF Data (2017, methodology): “For statistical signiﬁcance testing, we followed the bootstrapping algorithm described in (Koehn and Monz, 2006).”
Findings of the 2018 Conference on Machine Translation (WMT18) (2018, neutral): “This conference builds on twelve previous editions of WMT as workshops and conferences (Koehn and Monz, 2006; Callison-Burch et al., 2007, 2008, 2009, 2010, 2011, 2012; Bojar et al., 2013, 2014, 2015, 2016a, 2017).”
Findings of the 2017 Conference on Machine Translation (WMT17) (2017, methodology): “This conference builds on eleven previous editions of WMT as workshops and conference (Koehn and Monz, 2006; Callison-Burch et al., 2007, 2008, 2009, 2010, 2011, 2012; Bojar et al., 2013, 2014, 2015, 2016a).”
(Meta-) Evaluation of Machine Translation (2007, background): “Previous ACL Workshops on Machine Translation were more limited in scope (Koehn and Monz, 2005; Koehn and Monz, 2006).”
Learning Continuous Phrase Representations for Translation Modeling (2014, methodology): “First, we train an English to French system based on the data of WMT 2006 shared task (Koehn and Monz 2006).”
Results of the WMT15 Metrics Shared Task (2015, methodology): “For this reason, the Metrics Shared Task is held annually at the Work-shop of Statistical Machine Translation 1 , starting with Koehn and Monz (2006) and following up to Mach´aˇcek and Bojar (2014).”
Results of the WMT14 Metrics Shared Task (2014, methodology): “In both years BLEU performs best among baseline metrics, closely followed by CDER.”
Findings of the 2009 Workshop on Statistical Machine Translation (2009, background): “This paper presents the results of the shared tasks of the 2009 EACL Workshop on Statistical Machine Translation, which builds on three previous workshops (Koehn and Monz, 2006; Callison-Burch et al., 2007; Callison-Burch et al., 2008).”
Results of the WMT19 Metrics Shared Task: Segment-Level and Strong MT Systems Pose Big Challenges (2019, background): “The WMT Metrics Shared Task1 annually serves as a venue to validate the use of existing metrics (including baselines such as BLEU), and to develop new ones; see Koehn and Monz (2006) through Ma et al. (2018).”
Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation (2021, methodology): “The first WMT evaluation campaign (Koehn and Monz, 2006) used adequacy and fluency ratings on a 5-point scale acquired from participants as their main metric.”
Further Meta-Evaluation of Machine Translation (2008, background): “This paper presents the results the shared tasks of the 2008 ACL Workshop on Statistical Machine Translation, which builds on two past workshops (Koehn and Monz, 2006; Callison-Burch et al., 2007).”
BLEU Might Be Guilty but References Are Not Innocent (2020, background): “Such discrepancies have also been observed in the past, especially when comparing rule-based and statistical systems (Bojar et al., 2016b; Koehn and Monz, 2006; Callison-Burch et al., 2006).”