A Theory of Statistical Inference for Ensuring the Robustness of Scientific Results

By Beau Coker, Cynthia Rudin, Gary King. 2018

33 citations recorded by the source corpus.

Inference is the process of using facts we know to learn about facts we do not know. A theory of inference gives assumptions necessary to get from the former to the latter, along with a definition for and summary of the resulting uncertainty. Any one theory of inference is neither right nor wrong but merely an axiom that may or may not be useful. Each of the many diverse theories of inference can be valuable for certain applications. However, no existing theory of inference addresses the tendency to choose, from the range of plausible data analysis specifications consistent with prior evidence, those that inadvertently favor one’s own hypotheses. Because the biases from these choices are a growing concern across scientific fields, and in a sense the reason the scientific community was invented in the first place, we introduce a new theory of inference designed to address this critical problem. We introduce hacking intervals, which are the range of a summary statistic one may obtain given a class of possible endogenous manipulations of the data. Hacking intervals require no appeal to hypothetical data sets drawn from imaginary superpopulations. A scientific result with a small hacking interval is more robust to researcher manipulation than one with a larger interval and is often easier to interpret than a classical confidence interval. Some versions of hacking intervals turn out to be equivalent to classical confidence intervals, which means they may also provide a more intuitive and potentially more useful interpretation of classical confidence intervals. This paper was accepted by J. George Shanthikumar, big data analytics.

Subfields: Artificial Intelligence, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

All Models are Wrong, but Many are Useful: Learning a Variable’s Importance by Studying an Entire Class of Prediction Models Simultaneously (2018, background): “Given a descriptor φ, we now show a general result that allows creation of finite-sample CIs for the best performing models R( ).”
A study in Rashomon curves and volumes: A new perspective on generalization and model simplicity in machine learning (2019, background): “Rashomon sets: Rashomon sets have been used for various purposes [40, 18, 13], though the concept has been given different names.”
Variable Importance Clouds: A Way to Explore Variable Importance for the Set of Good Models (2019, background): “well as the partial dependence plot (PDP) Friedman (2001), partial leverage plot (PLP) Velleman and Welsch (1981), and partial importance (PI) Casalicchio et al.”
Veridical data science (2019, background): “This issue has been previously acknowledged in the modeling stage by (46), who derive “hacking intervals” to assess the range of a summary statistic optimized over a possible set of data and algorithm perturbations.”
Exploring the cloud of variable importance for the set of all good models (2020, background): “The closest work to ours considers extreme statistics of the Rashomon set without characterizing it 3,14 .”