Metrics for Evaluation of Word-level Machine Translation Quality Estimation
By Varvara Logacheva, Michal Lukasik, Lucia Specia. 2016
6 citations recorded by the source corpus.
The aim of this paper is to investigate suitable evaluation strategies for the task of word-level quality estimation of machine translation. We suggest various metrics to replace F 1 -score for the “BAD” class, which is currently used as main metric. We compare the metrics’ performance on real system outputs and synthetically generated datasets and suggest a reliable alternative to the F 1 -BAD score — the multiplication of F 1 -scores for different classes. Other metrics have lower discriminative power and are biased by unfair labellings.
Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Ondrej Bojar
- Philipp Koehn
- Lucia Specia
- Graham Neubig
- Barry Haddow
- Matt Post
- Rico Sennrich
- Felix Stahlberg
In Natural Language Processing
- Ondrej Bojar
- Philipp Koehn
- Lucia Specia
- Barry Haddow
- Christof Monz
- Yvette Graham
- Matt Post
- Graham Neubig
In Evaluation & Benchmarks
- Ondrej Bojar
- Lucia Specia
- Philipp Koehn
- Yvette Graham
- Matt Post
- Barry Haddow
- Christian Federmann
- Marco Turchi
Citations in the corpus
- Findings of the 2016 Conference on Machine Translation (2016, methodology): “was shown in (Logacheva et al., 2016c), this metric is not biased neither towards “pessimistic” nor to “optimistic” labellings, and is good at discriminating between different systems.”