Simulating human judgment in machine translation evaluation campaigns
By Philipp Koehn. 2012
22 citations recorded by the source corpus.
Subfields: Natural Language Processing, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Natural Language Processing
- Philipp Koehn
- Lucia Specia
- Ondrej Bojar
- Benjamin Van Durme
- Matt Post
- Barry Haddow
- Graham Neubig
- Rico Sennrich
In Evaluation & Benchmarks
- Philipp Koehn
- Ondrej Bojar
- Lucia Specia
- Matt Post
- Barry Haddow
- Benjamin Van Durme
- Keisuke Sakaguchi
- Christian Federmann
Citations in the corpus
- Findings of the 2014 Workshop on Statistical Machine Translation (2014, methodology): “Introduced for WMT13, the E XPECTED W INS has an intuitive score demonstrated to be accurate in ranking systems according to an underlying model of “relative ability” (Koehn, 2012a).”
- Efficient Elicitation of Annotations for Human Evaluation of Machine Translation (2014, methodology): “Models of “relative ability” (Koehn, 2012; Hop-kins and May, 2013) are a welcome addition to methods for inferring system rankings from human judgments.”
- Personalized Machine Translation: Predicting Translational Preferences (2015, methodology): “Expected (EXPT) This metric, proposed by Koehn (2012) and used by Bojar et al. (2013) in order to rank the participating systems in the WMT benchmark, compares the expected wins of the two systems.”
- Squibs: Evaluating Human Pairwise Preference Judgments (2015, background): “Koehn (2012) evaluates empirically the approaches of both Bojar et al. (2011) and Lopez (2012), with a focus on determining which systems are statistically distinguishable in terms of performance, defining confidence bounds for this purpose.”
- Findings of the 2013 Workshop on Statistical Machine Translation (2013, methodology): “In re-7 cent evaluation campaigns, we tweaked the metric and now arrived at a intuitive score that has been demonstrated to be accurate in ranking systems according to their true quality (Koehn, 2012).”
- Ten Years of WMT Evaluation Campaigns: Lessons Learnt (2016, methodology): “…have used Sentence Ranking as the sole method of automatic evaluation (refining it according to certain criticisms (Bojar et al., 2011; Lopez, 2012; Koehn, 2012)), but ongoing problems with reliability, interpretability and poor scalability with increasing numbers of systems have driven the search…”