Better Hypothesis Testing for Statistical Machine Translation: Controlling for Optimizer Instability

By J. Clark, Chris Dyer, Alon Lavie, Noah A. Smith. 2011

515 citations recorded by the source corpus.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

A Call for Clarity in Reporting BLEU Scores (2018, background): “The same is true of another evaluation framework, MultEval (Clark et al., 2011), which explicitly advocates for user-supplied tokenization.”
Can Neural Machine Translation be Improved with User Feedback? (2018, methodology): “For significance tests we used approximate randomization (Clark et al., 2011), for BLEU score evaluation (lowercased) the multi-bleu script of the Moses decoder (Koehn et al.”
Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning (2018, methodology): “Trained models are evaluated with respect to BLEU (Papineni et al., 2002), METEOR (Denkowski and Lavie, 2011) using MULTEVAL (Clark et al., 2011) and BEER (Stanojević and Sima’an, 2014) to cover a diverse set of automatic measures for translation quality.6 We test for statistical significance with…”
Achieving Human Parity on Automatic Chinese to English News Translation (2018, background): “3 Guarding Against Confounds Whenever trying to draw a conclusion based on a pair of different translations, we must avoid measuring the effects of extraneous variables that can confound the experimental variables we wish to measure [9].”
Comparing Automatic Evaluation Measures for Image Description (2014, methodology): “We collected the BLEU , TER , and Meteor scores using MultEval (Clark et al., 2011), and the ROUGE - SU 4 scores using the RELEASE-1.”
Getting Gender Right in Neural Machine Translation (2019, neutral): “(Clark et al., 2011).”
Graph Convolutional Encoders for Syntax-aware Neural Machine Translation (2017, methodology): “We report (cased) BLEU results (Papineni et al., 2002) using MultEval (Clark et al., 2011).”
Ask the Right Questions: Active Question Reformulation with Reinforcement Learning (2017, methodology): “tions each, of which we select a random one as the source and the other four as references. We use beam search, to compute the top hypothesis and report uncased, moses-tokenized BLEU using multeval9 (Clark et al., 2011). Please note, that the MSCOCO data is only used for evaluation purposes. Example”
SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translation (2018, methodology): “We also conduct pairwise statistical significance tests using paired bootstrap (Clark et al., 2011), and record the results in Table 1.”
A Shared Task on Multimodal Machine Translation and Crosslingual Image Description (2016, methodology): “We evaluated the sub-missions based on Meteor (Denkowski and Lavie, 2014) (primary), BLEU (Papineni et al., 2002) and TER (Snover et al., 2006) using MultEval (Clark et al., 2011) 11 with default parameters.”
Speech-to-Speech Translation Between Untranscribed Unknown Languages (2019, methodology): “We utilized two metrics to evaluate the translation performance from the transcribed text: BLEU scores [34] and METEOR [35] with a Multeval toolkit [36].”
Findings of the Third Shared Task on Multimodal Machine Translation (2018, methodology): “These are the top-scoring submissions and those that are not signiﬁ-cantly different (based on Meteor scores) according the approximate randomisation test (with p-value ≤ 0.05) provided by MultEval .”
Probing the Need for Visual Context in Multimodal Machine Translation (2019, methodology): “We train all systems three times each with different random initialization in order to perform significance testing with multeval (Clark et al., 2011).”
A Systematic Comparison of Smoothing Techniques for Sentence-Level BLEU (2014, methodology): “We perform random replications of parameter tuning, as suggested by Clark et al. (2011).”
Stronger Baselines for Trustable Results in Neural Machine Translation (2017, methodology): “Evaluation is conducted by average BLEU score over multiple independent training runs (Papineni et al., 2002; Clark et al., 2011).”
Sheffield MultiMT: Using Object Posterior Predictions for Multimodal Machine Translation (2017, methodology): “Evaluation was performed using Mul-tEval (Clark et al., 2011 The underlined systems were ofﬁcial submissions to the shared task. as described in the shared task website.”
UvA-DARE ( Digital Academic Repository ) Latent Domain Translation Models in Mix-of-Domains Haystack (2014, unknown): “The k-best batch MIRA optimizer (Cherry and Foster, 2012) was run at least three times to optimize any SMT system to avoid instability (Clark et al., 2011).6”
Generalizing Back-Translation in Neural Machine Translation (2019, result): “We report the statistical signiﬁcance of test2015 test2017 test2018 our results with MultEval (Clark et al., 2011).”