Neural Network Attributions: A Causal Perspective

By Aditya Chattopadhyay, Piyushi Manupriya, Anirban Sarkar, Vineeth Nallure Balasubramanian. 2019

168 citations recorded by the source corpus.

We propose a new attribution method for neural networks developed using first principles of causality (to the best of our knowledge, the first such). The neural network architecture is viewed as a Structural Causal Model, and a methodology to compute the causal effect of each feature on the output is presented. With reasonable assumptions on the causal structure of the input data, we propose algorithms to efficiently compute the causal effects, as well as scale the approach to data with large dimensionality. We also show how this method can be used for recurrent neural networks. We report experimental results on both simulated and real datasets showcasing the promise and usefulness of the proposed algorithm.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Explainable machine learning in deployment (2019, neutral): “tions [13].”
Causal Interpretability for Machine Learning - Problems, Methods and Evaluation (2020, methodology): “ualize the local decisions of the model by saliency map. Moreover, to further investigate how well the model estimates the ACE, they proposed to run the model on datasets for causal eﬀect estimations [12]. Causal Fairness Evaluation. Evaluation of causal fairness models is a challenging task. Papers”
XOC: Explainable Observer-Classifier for Explainable Binary Decisions (2019, background): “[5] establishes the connection between causality and network attribution and [33] propose a model agnostic way of training a causal explanation model that interprets a given predictive model.”
Causal Abstractions of Neural Networks (2021, methodology): “[6] develop an attribution method that explicitly treats neural models as structured causal models and directly computes the individual causal effect of a feature to determine its attribution.”
A historical perspective of explainable Artificial Intelligence (2020, background): “However, causal explanations are largely lacking in the machine learning literature, with only few exceptions such as (Chattopadhyay, Manupriya, Sarkar, & Balasubramanian, 2019).”
Feature relevance quantification in explainable AI: A causality problem (2019, methodology): “2016) for DeepLIFT, Binder et al. (2016) for Layer-wise Relevance Propagation (LRP), Ribeiro and Singh (2016) for Local Interpretable Modelagnostic Explanations (LIME), and for gradient based methods Chattopadhyay et al. (2019). For a summary of common architecture agnostic methods, see Molnar (2019”
CXPlain: Causal Explanations for Model Interpretation under Uncertainty (2019, unknown): “Besides these four established categories of feature importance estimation methods, structural causal models (SCMs) [29] and Deep Taylor Decomposition (DTD) [30] have also recently been proposed as explanation methods.”
Empowering Things With Intelligence: A Survey of the Progress, Challenges, and Opportunities in Artificial Intelligence of Things (2020, unknown): “interpretation methods [232] and example-based interpretation”
Explainable artificial intelligence (XAI) in deep learning-based medical image analysis (2021, background): “It would be of interest to incorporate such analyses in explanation of medical images, as Chattopadhyay et al. (2019) have done for visual explanation of MNIST data.”
Evaluating Explanation Without Ground Truth in Interpretable Machine Learning (2019, unknown): “of some input features [4].”
Epistemic values in feature importance methods: Lessons from feminist epistemology (2021, background): “approach, where they specify desirable properties for explanations, then construct or derive a method that has those properties [9, 13, 51, 68, 71, 76].”