Validation of an Automatic Metric for the Accuracy of Pronoun Translation (APT)
By Lesly Miculicich, Andrei Popescu-Belis. 2017
66 citations recorded by the source corpus.
In this paper, we define and assess a reference-based metric to evaluate the accuracy of pronoun translation (APT). The metric automatically aligns a candidate and a reference translation using GIZA++ augmented with specific heuristics, and then counts the number of identical or different pronouns, with provision for legitimate variations and omitted pronouns. All counts are then combined into one score. The metric is applied to the results of seven systems (including the baseline) that participated in the DiscoMT 2015 shared task on pronoun translation from English to French. The APT metric reaches around 0.993-0.999 Pearson correlation with human judges (depending on the parameters of APT), while other automatic metrics such as BLEU, METEOR, or those specific to pronouns used at DiscoMT 2015 reach only 0.972-0.986 Pearson correlation.
Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Ivan Titov
- Elena Voita
- Kellie Webster
- Emily Pitler
- Yonatan Belinkov
- M. Zhang
- Rico Sennrich
- Zhengxian Gong
In Natural Language Processing
- Ivan Titov
- Elena Voita
- Kellie Webster
- Emily Pitler
- Yonatan Belinkov
- M. Zhang
- Rico Sennrich
- Zhengxian Gong
In Evaluation & Benchmarks
Citations in the corpus
- Scalable Cross Lingual Pivots to Model Pronoun Gender for Translation (2020, background): “Specific to the translation of implicit or dropped pronouns, Wang et al. (2018) shows that modeling pronoun prediction jointly with translation improves quality, while Miculicich Werlen and Popescu-Belis (2017a) presents a proof-of-concept where coreference links are available at decoding time.”
- Context-Aware Neural Machine Translation Learns Anaphora Resolution (2018, background): “Ambiguous pronouns are relatively sparse in a general-purpose test set, and previous work has designed targeted evaluation of pronoun translation (Hardmeier et al., 2015; Miculicich Werlen and Popescu-Belis, 2017; Bawden et al., 2018).”
- Encouraging Lexical Translation Consistency for Document-Level Neural Machine Translation (2021, methodology): “We follow Miculicich et al. (2018) and Tan et al. (2019) to evaluate coreference and anaphora using the reference-based metric: accuracy of pronoun translation (Werlen and Popescu-Belis, 2017).”