Visualizing the Feature Importance for Black Box Models

By Giuseppe Casalicchio, Christoph Molnar, Bernd Bischl. 2018

219 citations recorded by the source corpus.

In recent years, a large amount of model-agnostic methods to improve the transparency, trustability and interpretability of machine learning models have been developed. We introduce local feature importance as a local version of a recent model-agnostic global feature importance method. Based on local feature importance, we propose two visual tools: partial importance (PI) and individual conditional importance (ICI) plots which visualize how changes in a feature affect the model performance on average, as well as for individual observations. Our proposed methods are related to partial dependence (PD) and individual conditional expectation (ICE) plots, but visualize the expected (conditional) feature importance instead of the expected (conditional) prediction. Furthermore, we show that averaging ICI curves across observations yields a PI curve, and integrating the PI curve with respect to the distribution of the considered feature results in the global feature importance. Another contribution of our paper is the Shapley feature importance, which fairly distributes the overall performance of a model among the features according to the marginal contributions and which can be used to compare the feature importance across different models.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, neutral): “Visual explanation Conditional / Dependence / Shapley plots: [56, 224, 235, 236]”
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI) (2018, methodology): “[113] proposed a local feature importance based approach that uses both partial importance (PI) and individual conditional importance (ICI) plots as visual tools.”
Mathematical optimization in classification and regression trees (2021, background): “2008) or on Game Theory concepts from cooperative games, such as the Shapley value (Casalicchio et al. 2019; Molnar et al. 2018).”
Variable Importance Clouds: A Way to Explore Variable Importance for the Set of Good Models (2019, methodology): “recent attempt to visualize variable importance is made by Casalicchio et al. (2018). They introduce a local variable importance measure and propose visualization tools to understand how changes in a feature affect model performance both on average and for individual data points.”
On Interpretability of Artificial Neural Networks: A Survey (2020, methodology): “Furthermore, Shapley value from cooperative game theory was used in [6], [27], [39], [113], [115].”
Trustworthy Artificial Intelligence: A Review (2022, background): “Example Based Local [140, 206] Global, Local [108, 109] Rule Based Global [88, 218] Global, Local [110] Visualization Based Global, Local [36, 209, 228]”
Explainable Artificial Intelligence: a Systematic Review (2020, methodology): “[176] proposed two alternatives to ICE plots, called Partial Importance (PI) and Individual Conditional Importance (ICI) plots, which visualize the feature importance rather than its prediction.”
Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence (2023, methodology): “Casalicchio et al. [134]. were able to break the dilemma between PDP and ICE by proposing an approach that could use both of the visualisation tools as an aid to make black-box models transparent.”
Exploring the cloud of variable importance for the set of all good models (2020, background): “Breiman’s variable importance measure for random forests 2 , which the VIC uses, as well as the partial dependence plot (PDP) 15 , partial leverage plot (PLP) 16 and partial importance 17 are four such examples.”
Interpretable Machine Learning - A Brief History, State-of-the-Art and Challenges (2020, background): “Permutation feature importance [28,16] is a popular importance measure, originally suggested for random forests [13].”
Heart disease prediction using supervised machine learning algorithms: Performance analysis and comparison (2021, background): “It is particularly applied in areas such as biomedicine and social sciences due to their simplicity and interpretability of feature ranking or risk analysis [44].”