Programs as Black-Box Explanations

By Sameer Singh, Marco Tulio Ribeiro, Carlos Guestrin. 2016

57 citations recorded by the source corpus.

Recent work in model-agnostic explanations of black-box machine learning has demonstrated that interpretability of complex models does not have to come at the cost of accuracy or model flexibility. However, it is not clear what kind of explanations, such as linear models, decision trees, and rule lists, are the appropriate family to consider, and different tasks and models may benefit from different kinds of explanations. Instead of picking a single family of representations, in this work we propose to use "programs" as model-agnostic explanations. We show that small programs can be expressive yet intuitive as explanations, and generalize over a number of existing interpretable families. We propose a prototype program induction method based on simulated annealing that approximates the local behavior of black-box classifiers around a specific prediction using random perturbations. Finally, we present preliminary application on small datasets and show that the generated explanations are intuitive and accurate for a number of classifiers.

Subfields: Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Safety, Robustness & Interpretability

Citations in the corpus

Local Rule-Based Explanations of Black Box Decision Systems (2018, methodology): “[33] adopts a simulated annealing approach that randomly grows, shrinks, or replaces nodes in an expression tree.”
A Survey of Methods for Explaining Black Box Models (2018, background): “Reference [110] is an antecedent of Anchor for tabular data only.”
Beyond Sparsity: Tree Regularization of Deep Models for Interpretability (2017, background): “In parallel, research efforts have emphasized that simple lists of (perhaps locally) important features are not sufﬁcient: Singh, Ribeiro, and Guestrin (2016) provide explanations in the form of programs; Lakkaraju, Bach, and Leskovec (2016) learn decision sets and show beneﬁts over other…”
Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations (2017, background): “Alternatively, explanations can be model-agnostic and relatively domain-general, as exemplified by LIME (Local Interpretable Model-agnostic Explanations, [Ribeiro et al., 2016; Singh et al., 2016]) which trains and presents local sparse models of how predictions change when inputs are perturbed.”
How do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-Interpretability of Explanation (2018, methodology): “…and Doshi-Velez, 2016, Hughes et al., 2016, Jung et al., 2017], various kinds of logic-based methods [Wang and Rudin, 2015, Lakkaraju et al., 2016, Singh et al., 2016, Liu and Tsang, 2016, Safavian and Landgrebe, 1991, Wang et al., 2017], techniques for extracting local explanations from…”
Human Evaluation of Models Built for Interpretability (2019, unknown): “Examples of human-simulatable models include human-simulatable regression functions (e.g. Caruana et al., 2015; Ustun and Rudin, 2016), and various logic-based methods (e.g. Wang and Rudin, 2015; Lakkaraju, Bach, and Leskovec, 2016; Singh, Ribeiro, and Guestrin, 2016).”
Questioning the AI: Informing Design Practices for Explainable AI User Experiences (2020, background): “to guarantee the prediction [38, 74, 83] Why, How to still be this”
On the Privacy Risks of Model Explanations (2019, methodology): “The LIME (Local Interpretable Model-agnostic Explanations) method [36] creates a local approximation of the model via sampling.”
An Evaluation of the Human-Interpretability of Explanation (2019, background): “, 2015], procedures [Singh et al., 2016], falling rule lists [Wang and Rudin, 2015], exemplars [Kim et al.”
Towards the Augmented Pathologist: Challenges of Explainable-AI in Digital Pathology (2017, background): “This belief is however challenged by recent work, in which carefully designed interpretation techniques have shed light on some of the most complex and deepest machine learning models (Singh et al., 2016).”
Accountability of AI Under the Law: The Role of Explanation (2017, unknown): “…an explanation of the system’s behavior overall, is known as local explanation [Ribeiro et al., 2016, Lei et al., 2016, Adler et al., 2016, Fong and Vedaldi, 2017, Selvaraju et al., 2016, Smilkov et al., 2017, Shrikumar et al., 2016, Kindermans et al., 2017, Ross et al., 2017, Singh et al., 2016].”