Interpreting Blackbox Models via Model Extraction

By Osbert Bastani, Carolyn Kim, Carolyn Kim, Hamsa Bastani. 2017

196 citations recorded by the source corpus.

Interpretability has become incredibly important as machine learning is increasingly used to inform consequential decisions. We propose to construct global explanations of complex, blackbox models in the form of a decision tree approximating the original model---as long as the decision tree is a good approximation, then it mirrors the computation performed by the blackbox model. We devise a novel algorithm for extracting decision tree explanations that actively samples new training points to avoid overfitting. We evaluate our algorithm on a random forest to predict diabetes risk and a learned controller for cart-pole. Compared to several baselines, our decision trees are both substantially more accurate and equally or more interpretable based on a user study. Finally, we describe several insights provided by our interpretations, including a causal issue validated by a physician.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

On the Art and Science of Machine Learning Explanations (2018, methodology): “This type of model is referred to by various other names, such as proxy or shadow models and the process of training surrogate models is sometimes referred to as model extraction [8], [30], [5].”
Interpreting tree ensembles with inTrees (2014, background): “, reduced decision nodes referred to as node harvest [28], rules generated from simplified predictive regions [22,23], single decision trees [3,4,34, 36], and linear models [14].”
Making deep neural networks right for the right scientific reasons by interacting with their explanations (2020, background): “Global approaches aim to explain the model by converting it as a whole to a more interpretable format [7],[47].”
Explanatory Interactive Machine Learning (2019, methodology): “the model by converting it as a whole to a more interpretable format [3, 4].”
Learning Interpretable Models with Causal Guarantees (2019, unknown): “In-deed, it has been shown that these issues often arise in practice, and that interpretability can help experts diagnose these issues (Caruana et al., 2015; Ribeiro et al., 2016; Bastani et al., 2017).”
A Survey on the Explainability of Supervised Machine Learning (2020, methodology): “The Decision Tree Extraction (Bastani et al., 2017) approach extracts a new model from a given black box model.”
Evaluating Explanation Methods for Neural Machine Translation (2020, methodology): “The mention of ﬁdelity is widely used in model compression (Buciluˇa et al., 2006; Polino et al., 2018), model distillation (Hinton et al., 2015; Liu et al., 2018), and particularly in evaluating the explanation models for black-box neural networks (Lakkaraju et al., 2016; Bastani et al., 2017).”
MAGIX: Model Agnostic Globally Interpretable Explanations (2017, methodology): “(Bastani et al., 2017) have used a surrogate model approach where they extract a decision tree that represents model behavior.”
Born-Again Tree Ensembles (2020, background): “Recently, some works focused on directly extracting a single tree from a tree ensemble based on stabilized but yet heuristic splitting criteria (Zhou & Hooker, 2016), genetic algorithms (Van-dewiele et al., 2017), or by actively sampling training points (Bastani et al., 2017a;b).”
Improving Simple Models with Confidence Profiles (2018, methodology): “There have been other strategies [9, 4, 6] to transfer information from bigger models to smaller ones, however, they are all similar in spirit to Distillation, where the complex models predictions are used to train a simpler model.”
Interpretable Machine Learning - A Brief History, State-of-the-Art and Challenges (2020, methodology): “Many IML methods are surrogate model approaches [89,75,72,95,34,10,18,61] and differ, e.”
LEMNA: Explaining Deep Learning based Security Applications (2018, background): “, decision tree [6] and decision set [31]) to incrementally approximate the target detection boundaries.”
TED: Teaching AI to Explain its Decisions (2018, methodology): “Creating a second, simpler-to-understand model, such as a small number of logical expressions, that mostly matches the decisions of the deployed model (e.g., (Bastani, Kim, and Bastani 2018; Caruana et al. 2015)).”