Show Your Work: Improved Reporting of Experimental Results

By Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, Noah A. Smith. 2019

310 citations recorded by the source corpus.

Research in natural language processing proceeds, in part, by demonstrating that new models achieve superior performance (e.g., accuracy) on held-out test data, compared to previous results. In this paper, we demonstrate that test-set performance scores alone are insufficient for drawing accurate conclusions about which model performs best. We argue for reporting additional details, especially performance on validation data obtained during model development. We present a novel technique for doing so: expected validation performance of the best-found model as a function of computation budget (i.e., the number of hyperparameter search trials or the overall training time). Using our approach, we find multiple recent model comparisons where authors would have reached a different conclusion if they had used more (or less) computation. Our approach also allows us to estimate the amount of computation required to obtain a given accuracy; applying it to several recently published results yields massive variation across papers, from hours to weeks. We conclude with a set of best practices for reporting experimental results which allow for robust future comparisons, and provide code to allow researchers to use our technique.

Subfields: Artificial Intelligence, General Social Sciences, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Social Sciences

In Evaluation & Benchmarks

Citations in the corpus

RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020, background): “Therefore, echoing calls for transparency in NLP research (Bender and Friedman, 2018; Mitchell et al., 2019; Dodge et al., 2019), we recommend researchers publicly release all relevant information during data collection (e.”
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “As a result of this, pretraining examples can be found in downstream test corpora (Dodge et al., 2021; Lee et al., 2021). Since it is impossible to remove the affected data from the training corpus after the release of a model, multiple approaches have been explored mitigation techniques. For exampl”
AI and the Everything in the Whole Wide World Benchmark (2021, background): “…et al., 2020, Ethayarajh and Jurafsky, 2020]; memory and compute requirements, which may be more or less constrained depending on the deployment context [Ethayarajh and Jurafsky, 2020, Dodge et al., 2019]; and stability in the face of perturbations to the training data [Sculley et al., 2018].”
Carbon Emissions and Large Neural Network Training (2021, background): “Estimates of Bitcoin’s impact for 2021 are ~78–121 TeraWatt-hours and ~37M–58M tCO 2 e [Cri21, Dig21].”
Understanding Dataset Difficulty with V-Usable Information (2021, neutral): “More-over, despite the performance of Transformer-based models varying across random initializations (Dodge et al., 2019; 2020; Mosbach et al., 2020), we find that PVI estimates are quite stable: the correlation across seeds is r > 0 .”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, background): “In Case 1 , for example, the researchers’ intention may be to advance the state of the art in the ﬁeld and allow other researchers to reproduce and build off of their work (Dodge et al., 2019).”
Guiding the Release of Safer E2E Conversational AI through Value Sensitive Design (2022, background): “In Case 1 , for example, the researchers’ intention may be to advance the state of the art in the field and allow other researchers to reproduce and build off of their work (Dodge et al., 2019).”
Energy and Policy Considerations for Modern Deep Learning Research (2020, background): “See Schwartz et al. (2019) for more discussion of standard metrics for reporting efﬁciency, and Dodge et al. (2019) for further analysis and concrete methods for reporting tuning and hyperparameter sensitivity.”
True Few-Shot Learning with Language Models (2021, methodology): “Inspired by prior work [77, 78], we offer recommendations for future work in true few-shot learning: • Report all hyperparameters (prompts) considered and the hyperparameter selection criteria.”
The Right Tool for the Job: Matching Model and Instance Complexities (2020, background): “(10)For increased reproduciblity (Dodge et al., 2019a), we also report validation results in Appendix B.”
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks (2020, methodology): “Adhering to the standards suggested by Dodge et al. (2019) for replication, we report our development set results in Tables 15, 17, and 18.”
Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping (2020, methodology): “These results demonstrate how model comparisons that only take into account reported performance in a benchmark can be misleading, and serve as a reminder of the value of more rigorous reporting practices (Dodge et al., 2019).”
Document Ranking with a Pretrained Sequence-to-Sequence Model (2020, methodology): “This procedure is consistent with the advice of Kaplan et al. [6] and recommendations by Dodge et al. [4], since we quantify effectiveness for a particular computational budget.”
Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022, background): “18653/v1/n19-1423 [42] Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A.”
What Will it Take to Fix Benchmarking in Natural Language Understanding? (2021, background): “Dodge et al. (2019) lay out a set of best practices for results reporting, with a focus on the impact of hyperparameter tuning on model comparison.”
Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics (2020, methodology): “Each experiment is run with 3 random seeds and a learning rate 22 chosen using the AllenTune package (Dodge et al., 2019).”