Interpretability via Model Extraction

By Osbert Bastani, Carolyn Kim, Hamsa Bastani. 2017

139 citations recorded by the source corpus.

The ability to interpret machine learning models has become increasingly important now that machine learning is used to inform consequential decisions. We propose an approach called model extraction for interpreting complex, blackbox models. Our approach approximates the complex model using a much more interpretable model; as long as the approximation quality is good, then statistical properties of the complex model are reflected in the interpretable model. We show how model extraction can be used to understand and debug random forests and neural nets trained on several datasets from the UCI Machine Learning Repository, as well as control policies learned for several classical reinforcement learning problems.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, positive): “Another contribution that falls oﬀ the same branch is that in [218] , where the authors formulate model simpliﬁcation as a model extraction process by approximating a transparent model to the complex one.”
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI) (2018, methodology): “[106] used a surrogate model approach where they extract a decision tree that represents model behavior.”
Interpreting Deep Classifier by Visual Distillation of Dark Knowledge (2018, background): “Knowledge distillation (Bucil˘a et al., 2006; Bastani et al., 2017; Ba & Caruana, 2014; Hinton et al., 2015) means training one model, called a student model , to generalize in the same way as a second teacher model , which is usually more complex.”
Interpreting tree ensembles with inTrees (2014, background): “, reduced decision nodes referred to as node harvest [28], rules generated from simplified predictive regions [22,23], single decision trees [3,4,34, 36], and linear models [14].”
Designing Interpretable Approximations to Deep Reinforcement Learning with Soft Decision Trees (2020, neutral): “In other areas such as flight control or self driving cars, DNNs are sidelined (at least for mass deployment) by their opaqueness or lack of decision making interpretability (Bastani, Kim, and Bastani 2017), (Hind et al. 2019).”
Toward Interpretable-AI Policies Using Evolutionary Nonlinear Decision Trees for Discrete-Action Systems (2020, background): “Bastani et al. [25] used axis-aligned DTs to develop interpretable models for black-box classiﬁers and RL-policies.”
"How do I fool you?": Manipulating User Trust via Misleading Black Box Explanations (2019, background): “An alternate approach is to provide a global explanation that describes the black box as a whole, typically summarizing it using an interpretable model [2, 12], which is useful in validating the black boxes before they are deployed to automatically make decisions (i.”
Verifiable Reinforcement Learning via Policy Extraction (2018, methodology): “There has also been work using model compression to learn decision trees [34, 7], but the focus has been on supervised learning rather than reinforcement learning, and on interpretability rather than veriﬁcation.”
Explaining Explanations in AI (2018, background): “A trade-off inherently occurs between the insightfulness of the approximated model, the simplicity of the presented function, and the size of the domain to which is applies and remains valid (Bastani et al., 2017; Lakkaraju et al., 2017).”
A survey of algorithmic recourse: definitions, formulations, solutions, and prospects (2020, background): “…above are prevalent not only in finance [20, 133] but also in justice (e.g., pretrial bail) [10], healthcare (e.g., prescription of life-altering medication) [24, 27, 72], and other settings (e.g., hiring) [43, 134, 160] broadly classified as consequential decision-making settings [20, 34, 45, 91].”
Techniques for interpretable machine learning (2018, methodology): “9]. Moreover, a DNN is utilized to train a decision tree which mimics the input-output function captured by the neural network so that the knowledge encoded in DNN is transferred to the decision tree [5]. To avoid the overtting of the decision tree, active learning is applied for training. These tec”
On Explaining Decision Trees (2020, methodology): “Moreover, there has been work on distilling or approximating complex ML models with (soft) decision trees [21,9,8,74,76,56,75].”
Interpretable Deep Learning under Fire (2018, background): “ders their application in security-sensitive domains [10,32,57]. These drawbacks have spurred intensive research on improving the DNN interpretability via providing explanations at either model-level [7,31,56,79] or instance-level [15,21,54,62]. For example,in Figure1(a),an attribution map highlight”
A framework for the extraction of Deep Neural Networks by leveraging public data (2019, methodology): “training data on which the model was originally trained) and exploring the explainability of proprietary ML models (e.g., by training an explainable substitute model such as a decision tree classiﬁer [10]). All three authors contributed equally. yAll three authors contributed equally. arXiv:1905.091”
Questioning the AI: Informing Design Practices for Explainable AI User Experiences (2020, methodology): “(Global) Decision tree approximation Approximate the model to an interpretable decision-tree [11, 47, 52] How, Why, Why not, What if”
Explainable Artificial Intelligence: a Systematic Review (2020, background): “Model Extraction [168] and Partition Aware Local Model (PALM) [169] utilize decision trees (DTs) to approximate complex models with the assumption that, as long as the approximation quality is good, the statistical properties of the complex model are reflected in the interpretable ones.”
Human-grounded Evaluations of Explanation Methods for Text Classification (2019, methodology): “This explanation method is based on model extraction (Bastani et al., 2017).”
Adversarial Removal of Demographic Attributes Revisited (2019, methodology): “…the full P AN 16 T WIT dataset, and perform feature analysis using uptraining: Us-( ing a transparent, linear model to approximate our deep model (Bastani et al., 2017), we train a logistic regression model with L1 regularization on the train set representations, relabeled by the predictions of…”
Principles and Practice of Explainable Machine Learning (2020, methodology): “ instances will be utilized, resulting in better local performance. Additional techniques to construct rules explaining a model’s decisions can be found in [82, 83]. In similar spirit, the authors of [6] formulate model simpliﬁcation as a model extraction process by approximating a complex model usi”