Winner's Curse? On Pace, Progress, and Empirical Rigor

By David Sculley, Jasper Snoek, Alexander B. Wiltschko, A. Rahimi. 2018

195 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

Citations in the corpus

Show Your Work: Improved Reporting of Experimental Results (2019, background): “Sculley et al. (2018) list examples of similar evaluation issues, and suggest encouraging stronger standards for empirical evaluation.”
Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure (2020, background): “ution is known as “technical debt”. Like financial debt, technical debt compounds, ultimately slowing down long term progress in pursuit of short term gains (sometimes leading to the “Winner’s Curse” [127]). Technical debt is equally relevant to datasets, in fact dataset dependency debts can be more”
Neural Text Summarization: A Critical Evaluation (2019, background): “e,Goyal et al.(2017) in visual question answering, andXian et al.(2017) in zero-shot image classiﬁcation. Comments on the general state of scholarship in the ﬁeld of machine learning were presented bySculley et al. (2018);Lipton and Steinhardt(2019) and references therein. 3 Datasets 3.1 Underconstr”
Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift (2019, methodology): “Contributions In the spirit of calls for more rigorous understanding of existing methods (Lipton and Steinhardt, 2018; Sculley et al., 2018; Rahimi and Recht, 2017), this paper provides a benchmark for evaluating uncertainty that focuses not only on calibration in the i.”
Troubling Trends in Machine Learning Scholarship (2018, neutral): “rningthe details ofthe hyper-parameter tuning without the distractions. Similar evaluation issues have been observed for deep reinforcement learning [30] and generative adversarial networks [51]. See [68] for more discussion of lapses in empirical rigor and resulting consequences. In contrast, many ”
The GAN Landscape: Losses, Architectures, Regularization, and Normalization (2018, background): “enalty. While each of these issues taken in isolation seems minor, they compound to produce a mist around existing results which introduces friction in practical applications and the research process [26]. 5 Related Work A recent large-scale study on GANs and VAEs was presented in [ 18 ]. The author”
A survey and critique of multiagent deep reinforcement learning (2018, background): “We believe that together with following the advice on how to design experiments and report results [197], the community would also benefit from reporting negative results [100,108,270,286] for carefully designed hypothesis and experiments.”
Evaluation Gaps in Machine Learning Practice (2022, background): “The structural incentives of the “competition mindset” encouraged by leaderboards can pose challenges to empirical rigor [153].”
Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards? (2021, background): “, 2020), and thoughtful consideration of their influence on the direction of research (Sculley et al., 2018; Dotan and Milli, 2020).”
Cross-Domain Modeling of Sentence-Level Evidence for Document Retrieval (2019, background): “Although Sculley et al. (2018) remind us that the goal of science is not wins , but knowledge, the latter requires ﬁrst establishing strong base-lines that accurately quantify proposed contributions.”
The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study (2019, background): “, 2018; Nagarajan & Kolter, 2018), and practical approaches to hyper-parameter optimization are often developed in an ad-hoc fashion (Sculley et al., 2018).”
The Benchmark Lottery (2021, background): “[Torralba and Efros, 2011, Armstrong et al., 2009, Machado et al., 2018, Sculley et al., 2018, Lin, 2019, Bowman and Dahl, 2021, Bouthillier et al., 2021, Recht et al., 2019, Lin et al., 2021, Balduzzi et al., 2018, Lipton and Steinhardt, 2018, Bouthillier et al., 2019, Vania et al., 2020, Mishra…”
The Neural Hype and Comparisons Against Weak Baselines (2019, background): “[13] wrote: “We observe that the rate of empirical advancement may not have been matched by consistent increase in the level of empirical rigor across the field as a whole.”