CAUSAL INTERPRETATIONS OF BLACK-BOX MODELS

By Qingyuan Zhao, Trevor J. Hastie. 2019

442 citations recorded by the source corpus.

Abstract The fields of machine learning and causal inference have developed many concepts, tools, and theory that are potentially useful for each other. Through exploring the possibility of extracting causal interpretations from black-box machine-trained models, we briefly review the languages and concepts in causal inference that may be interesting to machine learning researchers. We start with the curious observation that Friedman’s partial dependence plot has exactly the same formula as Pearl’s back-door adjustment and discuss three requirements to make causal interpretations: a model with good predictive performance, some domain knowledge in the form of a causal diagram and suitable visualization tools. We provide several illustrative examples and find some interesting and potentially causal relations using visualization tools for black-box models.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability, Causal Inference & Discovery

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

In Causal Inference & Discovery

Citations in the corpus

Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias (2020, background): “In recent work, Zhao and Hastie (2019) used mediation analysis for interpreting black-box models.”
"How do I fool you?": Manipulating User Trust via Misleading Black Box Explanations (2019, background): “Second, there has been recent work on capturing causal relationships between input features and black box predictions (Zhao and Hastie 2019; Wachter, Mittelstadt, and Rus-sell 2017).”
Causal Interpretability for Machine Learning - Problems, Methods and Evaluation (2020, background): “Traditional Interpretability Interpretable Models: [106], [103], [64], [102], [65], [105], [50], [13], [40] Causal Interpretability Model-based: [77], [38], [12], [8], [112], [81], [7], Example-based: [35], [45], [39], [101], [76], [86], [73], [61], [60], [33], [34] Post-hoc: [47], [89], [66], [93],”
Feature relevance quantification in explainable AI: A causality problem (2019, background): “…conditional distributions The main ideas of this paragraph can already be found in Datta et al. (2016) in more general and abstract form, see also Friedman (2001) and Zhao and Hastie (2019), but we want to rephrase them in a way that optimally prepares the reader to the below discussion.”
Explaining Explanations to Society (2019, methodology): “In this paper, we explicitly examine DNNs, even though there have been important developments in explanatory systems not tailored to DNNs, such as randomized importance [20], rule lists [19], partial dependence plots [9, 22, 37], Shapely scores [22, 31], and Bayesian case based models [15].”
Prediction and behavioral analysis of travel mode choice: A comparison of machine learning and logit models (2020, background): “Notably, partial dependence plots may reveal causal relationships if the machine-learning model is accurate and the domain knowledge supports the underlying causal structure (Zhao and Hastie, 2019).”
Reasons, Values, Stakeholders: A Philosophical Framework for Explainable Artificial Intelligence (2021, methodology): “Causal explanations can be obtained with methods such as causal modeling or approximate decision trees [37, 38, 39, 40].”
Investigating Gender Bias in Language Models Using Causal Mediation Analysis (2020, background): “In recent work, Zhao and Hastie (2019) used mediation analysis for interpreting black-box models.”
Machine learning and deep learning—A review for ecologists (2022, unknown): “Many researchers thus assume that ML and DL are unable to generate ecological understanding and can only be used as predictive tools (but see Zhao & Hastie, 2021 ).”