Statistical Significance Tests for Machine Translation Evaluation

By Philipp Koehn. 2004

1,912 citations recorded by the source corpus.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates (2018, positive): “05) from baselines with bootstrap resampling (Koehn, 2004).”
Findings of the 2016 Conference on Machine Translation (2016, methodology): “(2007) was calculated with the bootstrap test (Koehn, 2004).”
Findings of the 2015 Workshop on Statistical Machine Translation (2015, neutral): “For each submitted run, the statistical signiﬁ-cance of performance differences with respect to the baseline and the re-implementation of Simard et al. (2007) is calculated with the bootstrap test (Koehn, 2004).”
Neural Generation for Czech: Data and Baselines (2019, methodology): “01) according to bootstrap resampling (Koehn, 2004).”
BPE-Dropout: Simple and Effective Subword Regularization (2019, neutral): “(Statistical significance is computed via bootstrapping (Koehn, 2004).”
Sequence-to-Sequence Generation for Spoken Dialogue via Deep Syntax Trees and Strings (2016, methodology): “The difference is statistically significant at 99% level according to pairwise bootstrap resampling test (Koehn, 2004).”
Findings of the 2017 Conference on Machine Translation (WMT17) (2017, methodology): “(2007) was calculated with the bootstrap test (Koehn, 2004).”
Chameleons in Imagined Conversations: A New Approach to Understanding Coordination of Linguistic Style in Dialogs (2011, methodology): “In all figures in this paper, error bars represent standard error, estimated via bootstrap resampling (Koehn, 2004).”
Joint Language and Translation Modeling with Recurrent Neural Networks (2013, methodology): “We verified significance over the target-only model using paired bootstrap resampling (Koehn, 2004) over all test sets (7526 sentences) at the p < 0.”
Transformers without Tears: Improving the Normalization of Self-Attention (2019, methodology): “We also measure statistical signiﬁcance using bootstrap resampling (Koehn, 2004).”
Learning to Generate Pseudo-Code from Source Code Using Statistical Machine Translation (T) (2015, methodology): “To evaluate statistical significance, we performed a pairwise bootstrap test [36] for 10,000 sets of evaluation sentences randomly extracted from these results.”
Manual and Automatic Evaluation of Machine Translation between European Languages (2006, methodology): “Hence, we use the bootstrap resampling method described by Koehn (2004).”
A Context-aware Natural Language Generator for Dialogue Systems (2016, methodology): “Statistical significance at 99% level has been assessed using pairwise bootstrap resampling (Koehn, 2004).”
Accurate Evaluation of Segment-level Machine Translation Metrics (2015, background): “Although significance tests for evaluation of MT systems and document-level metrics have been identified (Koehn, 2004; Graham and Baldwin, 2014; Graham et al., 2014b), no such test has been proposed for segment-level metrics, and it is unfortunately common to conclude success without taking into acc”
RankME: Reliable Human Ratings for Natural Language Generation (2018, methodology): “In order to directly compare DA and RR, we generated overall system rankings based on our different methods, using pairwise bootstrap test at 95% confidence level (Koehn, 2004) to establish statistically significant differences.”
Results of the WMT19 Metrics Shared Task: Segment-Level and Strong MT Systems Pose Big Challenges (2019, methodology): “We employ bootstrap resampling (Koehn, 2004; Graham et al., 2014b) to estimate confidence intervals for our Kendall’s Tau formulation, and metrics with non-overlapping 95% confidence intervals are identified as having statistically significant difference in performance.”
Regularization techniques for fine-tuning in neural machine translation (2017, methodology): “Statistical significance on the concatenated test sets scores is determined via bootstrap resampling (Koehn, 2004).”