Confidence Estimation for Machine Translation

By John Blatz, Erin Fitzgerald, George F. Foster, Simona Gandrabur, Cyril Goutte, Alex Kulesza, A. Sanchís, Nicola Ueffing. 2004

463 citations recorded by the source corpus.

We present a detailed study of confidence estimation for machine translation. Various methods for determining whether MT output is correct are investigated, for both whole sentences and words. Since the notion of correctness is not intuitively clear in this context, different ways of defining it are proposed. We present results on data from the NIST 2003 Chinese-to-English MT evaluation.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Towards a Better Metric for Evaluating Question Generation Systems (2018, methodology): “The annotator scores were ﬁrst scaled between 0 to 1 using the normalization method in (Blatz et al., 2004), and the normalized scores were averaged to obtain the ﬁnal gold score.”
METEOR: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments (2007, methodology): “We use the normalization method described in (Blatz et al., 2003).”
FBK-UPV-UEdin participation in the WMT14 Quality Estimation shared-task (2014, methodology): “Word Lexicons (WL) We compute two different features based on statistical word lexicons (Blatz et al., 2004):”
Transductive learning for statistical machine translation (2007, methodology): “…by the length | t | of the target sentence: Conﬁdence Estimation: The conﬁdence estimation which we implemented follows the approaches suggested in (Blatz et al., 2003; Uefﬁng and Ney, 2007): The conﬁdence score of a target sentence t is calculated as a log-linear combination of phrase posterior…”
Learning What’s Easy: Fully Differentiable Neural Easy-First Taggers (2017, background): “The goal is to evaluate a translation system’s quality without access to reference translations (Blatz et al., 2004; Specia et al., 2013).”
The UU Submission to the Machine Translation Quality Estimation Task (2016, background): “Different types of systems, including system-dependent (glass-box) or system-independent (black-box), linguistically or statistically motivated features, have been explored (Blatz et al., 2004; Quirk, 2004; Specia et al., 2009).”
Using on-line available sources of bilingual information for word-level machine translation quality estimation (2015, background): “This has motivated a growing interest in the ﬁeld of MT quality estimation (Blatz et al., 2004; Specia et al., 2010; Specia and Soricut, 2013), which is the ﬁeld that focuses on developing techniques that allow to estimate the quality of the translation hypotheses produced by an MT system.”
A Re-examination of Machine Learning Approaches for Sentence-Level MT Evaluation (2007, methodology): “We normalize human judges’ scores following the process described by Blatz et al. (2003).”
Quality estimation for translation selection (2014, background): “Quality Estimation (QE) [Blatz et al., 2004, Specia et al., 2009] has several applications in the context of Machine Translation (MT), considering the use of translations for both inbound (e.g. gisting) and outbound (e.g. post-editing) purposes.”
Strategy-Based Technology for Estimating MT Quality (2015, methodology): “Many different confidence measures are investigated in(Blatz et al 2003).”
Using Machine Translation Evaluation Techniques to Determine Sentence-level Semantic Equivalence (2005, methodology): “(Blatz et al., 2004) used a variant of the WER score and the NIST score at the sentence level to assign correct-17 ness to translation candidates, by scoring them with respect to a reference set.”
Generating Counter Narratives against Online Hate Speech: Data and Strategies (2020, methodology): “The task of the reviewer is a sentence-level Con-ﬁdence Estimation (CE) similar to the one of Machine Translation (Blatz et al., 2004).”
Semi-supervised model adaptation for statistical machine translation (2007, methodology): “The confidence estimation which we implemented follows the approaches suggested in Blatz et al. (2003) and Ueffing and Ney (2007), where the confidence score of a target sentence t is calculated as a log-linear combination of several different sentence scores.”
Artificial error generation for translation-based grammatical error correction (2016, background): “Judging hypotheses without looking at the source or reference sentences is a distinct task, more similar to sentence quality estimation for MT output (Blatz et al., 2004; Specia et al., 2009).”
Active Learning for Interactive Neural Machine Translation of Data Streams (2018, methodology): “A common and effective way for measuring the uncertainty of a MT system is to use con-ﬁdence estimation (Gandrabur and Foster, 2003; Blatz et al., 2004; Uefﬁng et al., 2007).”
UGENT-LT3 SCATE System for Machine Translation Quality Estimation (2015, background): “Machine Translation (MT) Quality Estimation (QE) is the task of providing a quality indicator for unseen automatically translated sentences without relying on reference translations (Gandrabur & Foster, 2003; Blatz et al., 2004).”
Multi-level Translation Quality Prediction with QuEst++ (2015, background): “Word-level QE (Blatz et al., 2004; Luong et al., 2014) is seemingly a more challenging task where a quality label is to be produced for each target word.”