Automatic Reference-Based Evaluation of Pronoun Translation Misses the Point
By Liane Guillou, Christian Hardmeier. 2018
36 citations recorded by the source corpus.
We compare the performance of the APT and AutoPRF metrics for pronoun translation against a manually annotated dataset comprising human judgements as to the correctness of translations of the PROTEST test suite. Although there is some correlation with the human judgements, a range of issues limit the performance of the automated metrics. Instead, we recommend the use of semi-automatic metrics and test suites in place of fully automatic metrics.
Subfields: Artificial Intelligence, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- M. Zhou
- Mohit Bansal
- Ryan Cotterell
- Hal Daumé Iii
- Mrinmaya Sachan
- Yang Trista Cao
- Furu Wei
- Dongdong Zhang
In Natural Language Processing
- Hal Daumé Iii
- M. Zhou
- Yang Trista Cao
- Ryan Cotterell
- Mrinmaya Sachan
- Dongdong Zhang
- Shuming Ma
- Mohit Bansal
Citations in the corpus
- Scalable Cross Lingual Pivots to Model Pronoun Gender for Translation (2020, neutral): “For a discussion of the adequacy of this approach, see Guillou and Hardmeier (2018).”
- Toward Gender-Inclusive Coreference Resolution (2019, background): “In the case of machine translation, folk lore has it that getting pronouns “correct” does not really impact Bleu scores, which was recently confirmed (and alternatives proposed) by Guillou and Hardmeier [2018].”
- BlonDe: An Automatic Evaluation Metric for Document-level Machine Translation (2021, background): “…phenomena have been proposed (Hardmeier et al., 2015; Guillou and Hardmeier, 2016; Burchardt et al., 2017; Isabelle et al., 2017; Rios Gonzales et al., 2017; Müller et al., 2018; Bawden et al., 2018; Voita et al., 2019; Guillou and Hardmeier, 2018, inter alia ) for document-level MT.”