Unsupervised Quality Estimation for Neural Machine Translation
By Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco (Paco) Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, Lucia Specia. Transactions of the Association for Computational Linguistics, 2020
310 citations recorded by the source corpus.
Abstract Quality Estimation (QE) is an important component in making Machine Translation (MT) useful in real-world applications, as it is aimed to inform the user on the quality of the MT output at test time. Existing approaches require large amounts of expert annotated data, computation, and time for training. As an alternative, we devise an unsupervised approach to QE where no training or access to additional resources besides the MT system itself is required. Different from most of the current work that treats the MT system as a black box, we explore useful information that can be extracted from the MT system as a by-product of translation. By utilizing methods for uncertainty quantification, we achieve very good correlation with human judgments of quality, rivaling state-of-the-art supervised QE models. To evaluate our approach we collect the first dataset that enables work on both black-box and glass-box approaches to QE.
Subfields: Artificial Intelligence, Information Systems
View Paper: https://www.semanticscholar.org/paper/8ae3421420ec57d551ef2f524b48dcc78f9337fc
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sebastian Farquhar
- Yarin Gal
- Dan Friedman
- Adji Bousso Dieng
- Lorenz Kuhn
- Rahaf Aljundi
- Graham Neubig
- Richard Socher
In Information Systems
Citations in the corpus
- The Vendi Score: A Diversity Evaluation Metric for Machine Learning (2022, methodology): “In text, variants of this metric include pairwise-bleu (Shen et al., 2019) and d-lex-sim (Fomicheva et al., 2020), in which the similarity function is an n-gram overlap metric such as bleu (Papineni et al., 2002).”
- Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation (2023, methodology): “Normalised entropy reimplements single-model variant of Malinin & Gales (2020), lexical similarity measures the average Rouge-L in a sampled set of answers for a given question analogously to Fomicheva et al. (2020), entropy and p ( True ) reimplement Kadavath et al. (2022).”
- Uncertainty Estimation in Autoregressive Structured Prediction (2021, unknown): “Additionally, Fomicheva et al. (2020); Wang et al.”