With Little Power Comes Great Responsibility

By Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, Dan Jurafsky. 2020

157 citations recorded by the source corpus.

Despite its importance to experimental design, statistical power (the probability that, given a real effect, an experiment will reject the null hypothesis) has largely been ignored by the NLP community. Underpowered experiments make it more difficult to discern the difference between statistical noise and meaningful model improvements, and increase the chances of exaggerated findings. By meta-analyzing a set of existing NLP papers and datasets, we characterize typical power for a variety of settings and conclude that underpowered experiments are common in the NLP literature. In particular, for several tasks in the popular GLUE benchmark, small test sets mean that most attempted comparisons to state of the art models will not be adequately powered. Similarly, based on reasonable assumptions, we find that the most typical experimental design for human rating studies will be underpowered to detect small model differences, of the sort that are frequently studied. For machine translation, we find that typical test sets of 2000 sentences have approximately 75% power to detect differences of 1 BLEU point. To improve the situation going forward, we give an overview of best practices for power analysis in NLP and release a series of notebooks to assist with future power analyses.

Subfields: Artificial Intelligence, Information Systems, Statistics and Probability, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Statistics and Probability

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “This can partially be counteracted by employing experts to paraphrase existing references, thus creating a wider set of reference points to which a metric can compare, but this direction has not been explored for NLG. Similar problems exist in summarization, where the same references are often used ”
All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text (2021, methodology): “To have adequate power in our analyses (based on a power analysis with β = 0.8; Card et al., 2020), we had 130 different evaluators for each of the 6 task settings (3 domains × 2 models).”
Dynabench: Rethinking Benchmarking in NLP (2021, background): “, 2020) and often lack sufficient statistical power (Card et al., 2020), suggesting that, although they may be valu-”
Nearest Neighbor Machine Translation (2020, methodology): “Our work builds upon recent results showing the effectiveness of nearest neighbor methods in uncon-ditional language models (Khandelwal et al., 2020).”
What Will it Take to Fix Benchmarking in Natural Language Understanding? (2021, methodology): “requires two orders of magnitude more evaluation data (Card et al., 2020).”
Automatic Text Evaluation through the Lens of Wasserstein Barycenters (2021, background): “Keeping in mind the work by (Card et al., 2020) stressing the importance of the size of the dataset, we follow (Zhang et al., 2019) and choose to work with WMT16 and additionally report results on WMT15 that both offer over 500 sentences per language (in contrast to new versions (Bojar et al.,…”
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2022, methodology): “Also a human evaluation study without a large enough sample size can fail to find statistically significant results due to insufficient statistical power (Card et al., 2020).”
Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking (2021, background): “Even the choice of random seed can lead to substantially different results [15], and improvements on the state-of-the-art are often not statistically signiﬁcant [8].”
The MultiBERTs: BERT Reproductions for Robustness Analysis (2021, background): “Similarly to §3, authors in the NLP literature have recommended best practices for statistical testing (Koehn, 2004; Dror et al., 2018; Berg-Kirkpatrick et al., 2012; Card et al., 2020; Søgaard et al., 2014; Peyrard et al., 2021), many of which are based on existing tests to estimate the uncertainty”
The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation (2021, result): “tions, annotator agreement), a finding in line with prior reproducibility studies outside open-ended text generation (Card et al., 2020; Howcroft et al., 2020; van der Lee et al., 2021).”
News Summarization and Evaluation in the Era of GPT-3 (2022, background): “This size is chosen to give sufficient statistical power (Card et al., 2020) while keeping costs for GPT3-D2 evaluation low to enable others to compare on this subset.”
Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning (2021, background): “If a test set is too small to detect performance differences between two models, random variation in the test set scores can lead to misinterpreting one method as superior to another [16, 22].”
The Human Evaluation Datasheet 1.0: A Template for Recording Details of Human Evaluation Experiments in NLP (2021, unknown): “See, e.g., Card et al. (2020).”
Underreporting of errors in NLG output, and what to do about it (2021, background): “Ideally, the number of outputs should be based on a power analysis to provide meaningful comparisons (Card et al., 2020; van der Lee et al., 2021), but other considerations, such as time and budget, may be taken into account.”
CLIPScore: A Reference-free Evaluation Metric for Image Captioning (2021, methodology): “To our knowledge, this evaluation was ﬁrst conducted by Anderson et al. (2016) 2020) for an overview of statistical power in NLP).”
Scarecrow: A Framework for Scrutinizing Machine Text (2021, background): “While these approaches may be helpful— sometimes (Card et al., 2020)—at ranking systems, they do not give us insight into exactly which parts of a generation fall short, and why .”
What happens if you treat ordinal ratings as interval data? Human evaluations in NLP are even more under-powered than you think (2021, methodology): “We included NEM 1-2 and re NEM to compensate for the fact that two of the datasets used by Card et al. (2020) are not publicly available and to include reproduced ratings for the same outputs.”
Can Small and Synthetic Benchmarks Drive Modeling Innovation? A Retrospective Study of Question Answering Modeling Approaches (2021, methodology): “We only modify the number of training examples and use the original SQuAD development set for evaluation in all experiments; larger evaluation sets are critical for well-powered model comparisons (Card et al., 2020).”
The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation (2021, background): “When the test set is small, reliability in BLEU scores between models and inferred conclusions may be compromised (Card et al., 2020).”