Local Rule-Based Explanations of Black Box Decision Systems

By Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Dino Pedreschi, Franco Turini, Fosca Giannotti. 2018

498 citations recorded by the source corpus.

The recent years have witnessed the rise of accurate but obscure decision systems which hide the logic of their internal decision processes to the users. The lack of explanations for the decisions of black box systems is a key ethical issue, and a limitation to the adoption of machine learning components in socially sensitive and safety-critical contexts. %Therefore, we need explanations that reveals the reasons why a predictor takes a certain decision. In this paper we focus on the problem of black box outcome explanation, i.e., explaining the reasons of the decision taken on a specific instance. We propose LORE, an agnostic method able to provide interpretable and faithful explanations. LORE first leans a local interpretable predictor on a synthetic neighborhood generated by a genetic algorithm. Then it derives from the logic of the local interpretable predictor a meaningful explanation consisting of: a decision rule, which explains the reasons of the decision; and a set of counterfactual rules, suggesting the changes in the instance's features that lead to a different outcome. Wide experiments show that LORE outperforms existing methods and baselines both in the quality of explanations and in the accuracy in mimicking the black box.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

CERTIFAI: Counterfactual Explanations for Robustness, Transparency, Interpretability, and Fairness of Artificial Intelligence models (2019, methodology): “Other general methods on explainability, fairness and robustness have been described by [Guidotti et al., 2018b],[Binns, 2017], and [Akhtar and Mian, 2018] respectively.”
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI) (2018, neutral): “Examples include [102]–[104].”
A Survey of Methods for Explaining Black Box Models (2018, background): “Even though this sort of exception analysis is hardly performed, it can be more informative than the direct explanation, and it can also provide clues about the application domain [37, 85].”
The Dangers of Post-hoc Interpretability: Unjustified Counterfactual Explanations (2019, methodology): “[ Guidotti et al. , 2018a ] ) look for the L 2 (resp.”
An Interpretable Model with Globally Consistent Explanations for Credit Risk (2018, background): “they provide useful patterns, which has been a popular form of explanation in the state-of-the-art literature on model explanations [7, 4, 13].”
Explaining machine learning classifiers through diverse counterfactual explanations (2019, methodology): “[16] extend this approach by fitting a decisiontree classifier to approximate the non-linear model and then tracing the decision-tree paths to generate explanations.”
A survey of algorithmic recourse: definitions, formulations, solutions, and prospects (2020, background): “Finally, genetic-based approaches [183, 190] find solutions over different evolutions of candidate solutions according to various heuristics [22, 49, 74, 97, 162], and benefit from being model-/datatype-/norm-agnostic via only requiring query access to the model.”
Meaningful Explanations of Black Box AI Decision Systems (2019, methodology): “Adifferent stream of approaches, still in the XbD setting, focuses on the local behavior of a black box (Guidotti et al. 2018b), searching for an explanation of the decision made for a speciﬁc instance.”
Questioning the AI: Informing Design Practices for Explainable AI User Experiences (2020, background): “fits to guarantee the prediction [39, 75, 99] Why, How to still be this”
Counterfactual Explanations for Machine Learning: A Review (2020, methodology): “Finally, some approaches do not cast the goal into an optimization problem and solve it using heuristics [57, 91, 113, 67].”
Counterfactual Explanations and Algorithmic Recourses for Machine Learning: A Review (2020, unknown): “[126] have attempted this unification, as they first learn a local decision tree and then interpret the inversion of decision nodes of the tree as counterfactual explanations.”
Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications (2021, methodology): “Other methods that explain by building a local surrogate include LORE [58] and Anchors [148].”
Designing Theory-Driven User-Centric Explainable AI (2019, methodology): “We used state-of-the-art XAI facilities, such as SHAP [73] for attribution, LORE [34] for counterfactual rules, MOEA/D for sensitivity analysis [107] and implemented visualizations in Tableau.”
Globally-Consistent Rule-Based Summary-Explanations for Machine Learning Models: Application to Credit-Risk Evaluation (2019, unknown): “…focus on this research area and refer the reader to survey papers on interpretable and explainable machine learning to other bodies of work (Rudin et al. 2022, Došilović et al. 2018, Guidotti et al. 2018b, Arya et al. 2020, Tjoa and Guan 2020, Roscher et al. 2020, Burkart and Huber 2021).”
On Interpretability of Artificial Neural Networks: A Survey (2020, background): “Another proposal LOcal Rule-based Explanation (LORE) was from [64].”
Algorithmic Recourse: from Counterfactual Explanations to Interventions (2020, methodology): “In particular, [6, 31, 52] solve (1) using gradientbased optimization; [43, 49] employ mixed-integer linear program solvers to support mixed numeric/binary data; [38] use graph-based shortest path algorithms; [26] use a heuristic search procedure by growing spheres around the factual instance; [13, ”
Open the Black Box Data-Driven Explanation of Black Box Decision Systems (2018, methodology): “…baselines: pure global classification [8, 11], i.e., learning a decision tree on the whole training dataset, and the collection of all local explanations obtained applying the local explanators LORE and Anchor [20, 38] separately to each example in the training dataset (removing duplicate rules).”
A comparison of instance-level counterfactual explanation algorithms for behavioral and textual data: SEDC, LIME-C and SHAP-C (2019, methodology): “Lastly, the proposed methods can have different types of explanation such as a (set of) rule(s) [31, 53, 19], feature importance rankings [29, 44, 7, 51] or visual explanations [7, 46].”
Current Challenges and Future Opportunities for XAI in Machine Learning-Based Clinical Decision Support Systems: A Systematic Review (2021, methodology): “Intensive care phenotyping Used SHAP [79] for attribution, LORE [129] for counterfactual rules and multiobjective evolutionary algorithm based on decomposition (MOEA/D) for sensitivity analysis [130] Tabular Lee et al. [119] Alzheimer’s Disease (AD) Regional abnormalities in the brain space are…”