Meaningful Explanations of Black Box AI Decision Systems

By Dino Pedreschi, Fosca Giannotti, Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini. 2019

225 citations recorded by the source corpus.

Black box AI systems for automated decision making, often based on machine learning over (big) data, map a user’s features into a class or a score without exposing the reasons why. This is problematic not only for lack of transparency, but also for possible biases inherited by the algorithms from human prejudices and collection artifacts hidden in the training data, which may lead to unfair or wrong decisions. We focus on the urgent open challenge of how to construct meaningful explanations of opaque AI/ML systems, introducing the local-toglobal framework for black box explanation, articulated along three lines: (i) the language for expressing explanations in terms of logic rules, with statistical and causal interpretation; (ii) the inference of local explanations for revealing the decision rationale for a specific case, by auditing the black box in the vicinity of the target instance; (iii), the bottom-up generalization of many local explanations into simple global ones, with algorithms that optimize for quality and comprehensibility. We argue that the local-first approach opens the door to a wide variety of alternative solutions along different dimensions: a variety of data sources (relational, text, images, etc.), a variety of learning problems (multi-label classification, regression, scoring, ranking), a variety of languages for expressing meaningful explanations, a variety of means to audit a black box.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

The Twin-System Approach as One Generic Solution for XAI: An Overview of ANN-CBR Twins for Explaining Deep Learning (2019, background): “Some recent reviews of the literature have begun to comment on the state of current XAI solutions [Guidotti et al., 2018; Pedreschi et al., 2019].”
Bias in data‐driven artificial intelligence systems—An introductory survey (2020, methodology): “A third stream aims at bridging the local and the global ones by defining a strategy for combining local models in an incremental way (Pedreschi et al., 2019).”
On The Stability of Interpretable Models (2018, background): “They are also used to explain the global logic of inscrutable black-box machine learning models, such as neural networks, or the local logic of specific decisions taken by such black-boxes [1], [2].”
How Case-Based Reasoning Explains Neural Networks: A Theoretical Analysis of XAI Using Post-Hoc Explanation-by-Example from a Survey of ANN-CBR Twin-Systems (2019, background): “As Pedreschi et al. [22] point out “the state of the art to date still exhibits ad-hoc , scattered results, mostly hardwired to specific models…[and]… a widely applicable, systematic approach has not emerged yet”.”
MultiVerse: Causal Reasoning using Importance Sampling in Probabilistic Programming (2019, background): “Counterfactuals are probabilistic in nature and difficult to infer, but are powerful for explanation (Wachter et al., 2017; Sokol and Flach, 2018; Guidotti et al., 2018; Pedreschi et al., 2019), fairness Kusner et al.”
Explainable artificial intelligence: an analytical review (2021, methodology): “(cid:129) Usage (post hoc, e.g., surrogate models (Dieber & Kirrane, 2020; Pedreschi et al., 2019; Tritscher et al., 2020) and intrinsic to the model architecture, e.g., explainable-by-design (Soares, Angelov, Biaso, et al., 2020; Soares, Angelov, Costa, et al., 2020)).”