Interpretability of deep learning models: A survey of results

By Supriyo Chakraborty, Richard J. Tomsett, Ramya Raghavendra, Daniel Harborne, Moustafa Farid Alzantot, Federico Cerutti, Mani B. Srivastava, Alun David Preece, Simon Justin Julier, R. Rao and 5 more. 2017

469 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Time-series forecasting with deep learning: a survey (2020, background): “k architectures, a new body of research has emerged in methods for interpreting deep learning models. We present a summary below – referring the reader to dedicated surveys for more in-depth analyses [69,70]. Techniques for Post-hoc Interpretability Post-hoc interpretable models are developed to int”
Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI (2019, background): “(Alonso, Castiello, and Mencar, 2018; Biran and Cotton, 2017; Brézillon, 1994; Brézillon and Pomerol, 1997; Buchanan and Shortliffe, 1984; Chakraborty et al., 2017; Clancey, 1986; Core et al., 2006; Craven and Shavlik, 1996; Doran, Shulz, and Besold, 2017; Doshi-Velez and Kim, 2017ab; Doyle,…”
Neural Prototype Trees for Interpretable Fine-grained Image Recognition (2020, background): “The tree can be understood in its entirety by users [34] and every step of the reasoning process can be reproduced [7].”
This Looks Like That, Because ... Explaining Prototypes for Interpretable Image Recognition (2020, background): “Intrinsically interpretable models on the other hand, are faithful by design and allow simulatability: a user should be able to reproduce the model’s decision making process based on the input data together with the explanations of the interpretable model and come to the same prediction [9, 28].”
NILE : Natural Language Inference with Faithful Natural Language Explanations (2020, background): “Typically, these models aim at improving label accuracy, while it is often desirable to also produce explanations for these decisions (Lipton, 2016; Chakraborty et al., 2017).”
DARPA's Explainable Artificial Intelligence (XAI) Program (2019, background): “As their use has increased, so has research on explainability from the perspectives of ML (Chakraborty et al. 2017; Ras et al. 2018) and cognitive psychology (Miller 2017).”
Using deep learning to model the hierarchical structure and function of a cell (2018, background): “A limitation of post hoc interpretation is that it is disconnected from training, leaving no guarantees as to what level of human understanding can be achieved 41 .”
Real-world data: a brief review of the methods, applications, challenges and opportunities (2022, background): “Promising and encouraging research work on this topic has already started [106–111], but more research is warranted.”
Definitions, methods, and applications in interpretable machine learning (2019, background): “One line of work focuses on providing an overview of different interpretation methods with a strong emphasis on post hoc interpretations of deep learning models (7, 8), sometimes pointing out similarities between various methods (9, 10).”
Free-Lunch Saliency via Attention in Atari Agents (2019, background): “Unfortunately, both RL and deep learning in general suffer from poor interpretability: it is hard to explain why a deep neural network performs as it does, hard to understand why an RL agent has assigned a specific value to a given state, and harder yet when these two inherently “black-box” methods ”
Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure (2020, background): “y “superstars” [60], “savants” [116] and saviours with brilliant innovative insights. Hacker culture emphasizes moving fast and debugging as errors arise [91] (neural network interpretability methods [31] are the stack trace debuggers of ML). This culture devalues the incremental and cooperative car”
Code Prediction by Feeding Trees to Transformers (2020, background): “Interpretability studies [36]–[39] that provide human-understandable justifications, have thus become an important property in building safe and trustworthy AI systems.”
Post-hoc Interpretability for Neural NLP: A Survey (2021, result): “In contrast to other surveys and tutorials on interpretability methods [14, 15, 17, 25, 35, 80, 114, 120, 126] which only discusses the most popular approaches (usually 3 among input features, adversarial examples, influential examples, projection, and linguistic information), this survey offers a m”
Interpretable Convolutional Filters with SincNet (2018, background): “In recent years, a growing interest has been thus devoted to the development of interpretable machine learning [6, 7], as witnessed by the numerous works in the field, ranging from visualization [8, 9], diagnosis of DNNs [10], explanatory graphs [11], and explainable models [12], just to name a few.”