Model-Agnostic Interpretability of Machine Learning

By Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin. 2016

964 citations recorded by the source corpus.

Understanding why machine learning models behave the way they do empowers both system designers and end-users in many ways: in model selection, feature engineering, in order to trust and act upon the predictions, and in more intuitive user interfaces. Thus, interpretability has become a vital concern in machine learning, and work in the area of interpretable models has found renewed interest. In some applications, such models are as accurate as non-interpretable ones, and thus are preferred for their transparency. Even when they are not accurate, they may still be preferred when interpretability is of paramount importance. However, restricting machine learning to interpretable models is often a severe limitation. In this paper we argue for explaining machine learning predictions using model-agnostic approaches. By treating the machine learning models as black-box functions, these approaches provide crucial flexibility in the choice of models, explanations, and representations, improving debugging, comparison, and interfaces for a variety of users and models. We also outline the main challenges for such methods, and review a recently-introduced model-agnostic explanation approach (LIME) that addresses these challenges.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Measuring Attribution in Natural Language Generation Models (2021, neutral): “Ribeiro et al. (2016)).”
Editing Factual Knowledge in Language Models (2021, background): “One cannot easily access and interpret their computation and memories (Ribeiro et al., 2016; Be-linkov and Glass, 2019; Voita et al., 2019; De Cao et al., 2020), thus, modifying their knowledge is a challenging problem.”
Explaining Predictions from Tree-based Boosting Ensembles (2019, background): “e obvious advantage of model-agnostic explanations is that they can be applied to any type of model [9], but since the explanation is based on a local approximation of the original model, there exists some inherent degree of error between the original model and the local approximation.”
Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI (2019, neutral): “…and Francis, 2017; Kim, 2018; Kim, Shah, and Doshi-Velez, 2015; Kim Lin Collins, Taylor, and Amer, 2016; Koh and Liang, 2017; Kononenko, et al., 2013; Kulesza, Burnett, Wong, and Stumpf, 2015; Park et al., 2016; Rajani and Mooney, 2017; Ribeiro, Singh, and Guestrin, 2016b, 2016a; Ross, Hughes, and”
Anchors: High-Precision Model-Agnostic Explanations (2018, methodology): “In Ribeiro, Singh, and Guestrin (2016b), we emphasize that the perturbations D (and explanations) must use an interpretable representation (i.e. one that makes sense to humans), even if the model uses an alternative representation of the input.”
Visualizing the Feature Importance for Black Box Models (2018, background): “Methods for feature effects include partial dependence (PD) plots [10], individual conditional expectation (ICE) plots [11], LIME [18] and, more recently, SHAP values [17].”
Explanations of model predictions with live and breakDown packages (2018, methodology): “An interesting example of a situation in which trust issues are fully justiﬁed is presented in Tulio Ribeiro et al. (2016b).”
What do we need to build explainable AI systems for the medical domain? (2017, background): “The explanation model is g : R d → R , g ∈ G , where G is a class of potentially interpretable models, such as linear models, decision trees, or rule lists; given a model g ∈ G , it can be visualized as an explanation to the human expert (for details please refer to (Ribeiro et al., 2016a)).”
Interpretable Predictions of Tree-based Ensembles via Actionable Feature Tweaking (2017, methodology): “…space; (ii) We ϵ change that each feature can sustain; (iii) We experiment with five δ (iv) We leverage on the importance of each feature derived from the model to rank the final list of recommendations; (v) We focus More recent work on related topics are those of Ribeiro et al. [20, e.g. e.g.”
Nothing Else Matters: Model-Agnostic Explanations By Identifying Prediction Invariance (2016, background): “Model-agnostic explanations [12] avoid the need to trade off accuracy by treating the model as a black box.”
TreeView: Peeking into Deep Neural Networks Via Feature-Space Partitioning (2016, background): “Also, the recently developed locally interpretable modelagnostic explanations (LIME) approach poses interpretation as a sparse local fitting problem [3].”
Causal Interpretability for Machine Learning - Problems, Methods and Evaluation (2020, methodology): “Traditional Interpretability Interpretable Models: [106], [103], [65], [102], [66], [105], [51], [13], [40] Causal Interpretability Model-based: [77], [38], [12], [8], [112], [81], [7], Example-based: [35], [46], [39], [101], [76], [86], [73], [62], [61], [33], [34] Post-hoc: [48], [89], [67], [93],”
Explaining Classifiers with Causal Concept Effect (CaCE) (2019, background): “Many approaches have been proposed to tackle this challenge: for example by developing a model with interpretable components built-in [8, 16] or by building post-training interpretability methods [13, 14, 4, 2, 1, 9].”
Bias in medical AI: Implications for clinical decision-making (2024, methodology): “Popular interpretability methods for machine learning models include Shapley Additive Explanations (SHAP) and Local Interpretable Model-agnostic Explanations (LIME) [102,103].”