Faithful and Customizable Explanations of Black Box Models

By Himabindu Lakkaraju, Ece Kamar, R. Caruana, Jure Leskovec. 2019

305 citations recorded by the source corpus.

As predictive models increasingly assist human experts (e.g., doctors) in day-to-day decision making, it is crucial for experts to be able to explore and understand how such models behave in different feature subspaces in order to know if and when to trust them. To this end, we propose Model Understanding through Subspace Explanations (MUSE), a novel model agnostic framework which facilitates understanding of a given black box model by explaining how it behaves in subspaces characterized by certain features of interest. Our framework provides end users (e.g., doctors) with the flexibility of customizing the model explanations by allowing them to input the features of interest. The construction of explanations is guided by a novel objective function that we propose to simultaneously optimize for fidelity to the original model, unambiguity and interpretability of the explanation. More specifically, our objective allows us to learn, with optimality guarantees, a small number of compact decision sets each of which captures the behavior of a given black box model in unambiguous, well-defined regions of the feature space. Experimental evaluation with real-world datasets and user studies demonstrate that our approach can generate customizable, highly compact, easy-to-understand, yet accurate explanations of various kinds of predictive models compared to state-of-the-art baselines.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI (2019, methodology): “nts 7 X Representer theorem [129] 7 X Sensitivity SocRat (Structured-output Causual Rationalizer) [126] 7 X Meta-predictors [125] 7 X Explanation vector [127] 7 7 # Also listed elsewhere: [14], [42], [84], [93] N.A. N.A. # Also listed elsewhere: [14], [59], [84] etc N.A. N.A. Optimization CNN with s”
"How do I fool you?": Manipulating User Trust via Misleading Black Box Explanations (2019, methodology): “normal, non-negative, non-monotone, and submodular with matroid constraints [12].”
Manipulating and Measuring Model Interpretability (2018, background): “2 RELATEDWORK Although there has been a recent surge of research in the machine learning community on techniques for achieving interpretability [15, 48, 56, 61, 63, 64, 74, 89, 93, 98, 102, 103, 108], there have been relatively fewer studies of how factors commonly thought to make machine learning m”
Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? (2020, methodology): “A more direct application of the Model Assumption is via the notion of fidelity (Guidotti et al., 2018; Lakkaraju et al., 2019).”
A Survey on the Explainability of Supervised Machine Learning (2020, background): “Model Understanding through Subspace Explanations (MUSE) (Lakkaraju et al., 2019) is a model-agnostic framework that provides understanding of a black box model by explaining how it behaves in the sub-spaces defined by the features of interest.”
Interpretability of machine learning‐based prediction models in healthcare (2020, methodology): “In the future, we expect more approaches like MUSE (Lakkaraju et al., 2019) where a global interpretability approach is supplemented by specific interpretations that can explain predictions either for a single individual or a smaller group of individuals.”
No Explainability without Accountability: An Empirical Study of Explanations and Feedback in Interactive ML (2020, unknown): “, differential explanations [37]) might be equivalently faithful, while better counteracting the user experience concerns.”
Explainable Artificial Intelligence Applications in Cyber Security: State-of-the-Art in Research (2022, methodology): “[71] proposed a model-agnostic technique called Model Understanding through Subspace Explanations (MUSE), aiming at learning the behavious of a specific black-box model by yielding a small number of tight decision sets.”
On Relating Explanations and Adversarial Examples (2019, background): “Explanations can be local [46, 35, 18] or global [47, 29], depending on whether they only apply to a local neighborhood of a target instance or not.”
Unbox the black-box for the medical explainable AI via multi-modal and multi-centre data fusion: A mini-review, two showcases and beyond (2021, methodology): “…decision sets or rule sets have been studied for interpretability [159] and Model Understanding through Subspace Explanations (MUSE) method [160] has been developed to describe the projections of the global model by considering the various subgroups of instances defined by user interesting…”
RL-LIM: Reinforcement Learning-based Locally Interpretable Modeling (2019, methodology): “To be useful in practice, locally interpretable models need to maximize two objectives: (i) the overall prediction performance (how well it predicts compared to the ground truth labels) – for the model to be accurate, and (ii) fidelity (how well it approximates the ‘black-box’ model predictions) – t”