All Models are Wrong, but Many are Useful: Learning a Variable’s Importance by Studying an Entire Class of Prediction Models Simultaneously

By Aaron Fisher, Cynthia Rudin, Francesca Dominici. 2018

1,742 citations recorded by the source corpus.

Variable importance (VI) tools describe how much covariates contribute to a prediction model’s accuracy. However, important variables for one well-performing model (for example, a linear model f (x) = xT β with a fixed coefficient vector β) may be unimportant for another model. In this paper, we propose model class reliance (MCR) as the range of VI values across all well-performing model in a prespecified class. Thus, MCR gives a more comprehensive description of importance by accounting for the fact that many prediction models, possibly of different parametric forms, may fit the data well. In the process of deriving MCR, we show several informative results for permutation-based VI estimates, based on the VI measures used in Random Forests. Specifically, we derive connections between permutation importance estimates for a single prediction model, U-statistics, conditional variable importance, conditional causal effects, and linear model coefficients. We then give probabilistic bounds for MCR, using a novel, generalizable technique. We apply MCR to a public data set of Broward County criminal records to study the reliance of recidivism prediction models on sex and race. In this application, MCR can be used to help inform VI for unknown, proprietary models.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

In Safety, Robustness & Interpretability

Citations in the corpus

Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, background): “Our treatment of underspecification is more closely related to work on “Rashomon sets” (Fisher et al., 2019), “predictive multiplicity” (Marx et al.”
Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, methodology): “…assessment of fairness and variable importance (Rudin et al., 2020; Dimanov et al., 2020), and uncertainty bands for variable importance (Gosiewska and Biecek, 2020; Fisher et al., 2019), as well as overall difficulty troubleshooting the combination of a black box and an explanation on top of it.”
Forward stability and model path selection (2021, neutral): “Within the recent Rashomon literature are methods for decision making (Tulabandhula and Rudin 2013, 2014), identifying ranges of variable importance within a model class (Fisher et al. 2019), and measuring model class simplicity (Semenova and Rudin 2019).”
Characterizing Fairness Over the Set of Good Models Under Selective Labels (2021, methodology): “When defined in this manner, the set of good models is also called the “Rashomon set” (Rudin, 2019; Fisher et al., 2019; Dong & Rudin, 2020; Semenova et al., 2020).”
The explanation game: a formal framework for interpretable machine learning (2020, neutral): “Subsequent studies using the same dataset reached different conclusions regarding the algorithm’s reliance on race (Fisher et al. 2019; Rudin et al. 2018), high- lighting the inherent uncertainty of model reconstruction when the target algorithm is not forthcoming.”
Interpretable Machine Learning - A Brief History, State-of-the-Art and Challenges (2020, methodology): “ but also its explanations, are computed from data and hence are subject to uncertainty. First research is working towards quantifying uncertainty of explanations, for example, for feature importance [124,28,4], layer-wise relevance propagation [24], and Shapley values [127]. 6 Surrogate models are ”
Practical guide to SHAP analysis: Explaining supervised machine learning model predictions in drug development (2024, neutral): “48 Considering these alternatives may be worthwhile to address some of the SHAP limitations.”
A study in Rashomon curves and volumes: A new perspective on generalization and model simplicity in machine learning (2019, methodology): “Rashomon sets: Rashomon sets have been used for various purposes (Breiman et al., 2001; Srebro et al., 2010; Fisher et al., 2019; Coker et al., 2018; Tulabandhula and Rudin, 2014b; Meinshausen and Bühlmann, 2010; Letham et al., 2016; Nevo and Ritov, 2017).”
Exploring the cloud of variable importance for the set of all good models (2020, background): “With the expressions for both the Rashomon set (equation (1)) and the model reliance function (equation (3)), Theorem 1 in the main text follows naturally.”
SegNBDT: Visual Decision Rules for Segmentation (2020, background): “In general, shuffling values to ascertain feature importance is not new: this is known as permutation feature importance [2] or model reliance [5]; we simply apply this to images in pixel-space.”
Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies (2021, background): “[44] to compute global feature importance in their paper, exploring whether such explanations are helpful during decision making tasks.”
Directional Bias Amplification (2021, background): “Thus, like previous work (Fisher et al., 2019), we urge transparency, and advocate for the inclusion of confidence intervals.”