"How do I fool you?": Manipulating User Trust via Misleading Black Box Explanations

By Himabindu Lakkaraju, Osbert Bastani. 2019

298 citations recorded by the source corpus.

As machine learning black boxes are increasingly being deployed in critical domains such as healthcare and criminal justice, there has been a growing emphasis on developing techniques for explaining these black boxes in a human interpretable manner. There has been recent concern that a high-fidelity explanation of a black box ML model may not accurately reflect the biases in the black box. As a consequence, explanations have the potential to mislead human users into trusting a problematic black box. In this work, we rigorously explore the notion of misleading explanations and how they influence user trust in black box models. Specifically, we propose a novel theoretical framework for understanding and generating misleading explanations, and carry out a user study with domain experts to demonstrate how these explanations can be used to mislead users. Our work is the first to empirically establish how user trust in black box models can be manipulated via misleading explanations.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, background): “In particular, there are many serious problems with the use of explaining black boxes posthoc, as outlined in several papers that have shown why explaining black boxes can be misleading and why explanations do not generally serve their intended purpose (Rudin, 2019; Laugel et al., 2019; Lakkaraju an”
Do Not Trust Additive Explanations (2019, background): “This causes an increasing number of voices to avoid using explanations in high-stakes decisions (Rudin, 2019; Lakkaraju and Bastani, 2020; Loyola-González, 2019).”
A survey of algorithmic recourse: definitions, formulations, solutions, and prospects (2020, background): “1) may allow for an organization to cherry-pick “the most socially acceptable explanation out of many equally plausible ones” [21, 79, 99] (see also, fairwashing [6]).”
A Systematic Literature Review of User Trust in AI-Enabled Systems: An HCI Perspective (2022, background): “When poorly designed or adapted to target users, AI usage could mislead users into unfair and even incorrect decision-making (Lakkaraju & Bastani, 2020).”
Debugging Tests for Model Explanations (2020, background): “Feature attributions can be easily manipulated, providing evidence for a collective ‘weakness’ of current approaches [23, 24, 35, 61].”
Regularizing Black-box Models for Improved Interpretability (2019, background): “Note that we do not ask participants to rate their trust in the model because of issues such as those raised in Lakkaraju and Bastani [2020].”