I Think I Get Your Point, AI! The Illusion of Explanatory Depth in Explainable AI

By Michael Chromik, Malin Eiband, Felicitas Buchner, Adrian Krüger, Andreas Butz. 2021

165 citations recorded by the source corpus.

Unintended consequences of deployed AI systems fueled the call for more interpretability in AI systems. Often explainable AI (XAI) systems provide users with simplifying local explanations for individual predictions but leave it up to them to construct a global understanding of the model behavior. In this work, we examine if non-technical users of XAI fall for an illusion of explanatory depth when interpreting additive local explanations. We applied a mixed methods approach consisting of a moderated study with 40 participants and an unmoderated study with 107 crowd workers using a spreadsheet-like explanation interface based on the SHAP framework. We observed what non-technical users do to form their mental models of global AI model behavior from local explanations and how their perception of understanding decreases when it is examined.

Subfields: Artificial Intelligence, Management Science and Operations Research, Safety Research, Safety, Robustness & Interpretability, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Q. Liao
2. Kush R. Varshney
3. Microsoft Research
4. Richa Varshney
5. Canada Kush
6. Rick Salay
7. Krzysztof Czarnecki
8. Cynthia Rudin

In Management Science and Operations Research

1. Q. Liao
2. Kush R. Varshney

In Safety Research

1. Kush R. Varshney
2. Richa Varshney
3. Canada Kush
4. Microsoft Research
5. Q. Liao

In Safety, Robustness & Interpretability

1. Q. Liao
2. Kush R. Varshney
3. Richa Varshney
4. Canada Kush
5. Microsoft Research

In Human-Computer Interaction

1. Q. Liao
2. Microsoft Research

Citations in the corpus

• Human-Centered Explainable AI (XAI): From Algorithms to User Experiences (2021, background): “In addition, there is the concern of illusory understanding, with which one subjectively over-estimates the understanding they gain from XAI [22].”
• Artificial intelligence and illusions of understanding in scientific research (2024, unknown): “Recent work suggests that this illusion inflates user understandings of AI systems 109 .”