Explaining Decision-Making Algorithms through UI: Strategies to Help Non-Expert Stakeholders

By Hao Fei Cheng, Ruotong Wang, Zheng Zhang, Fiona O'Connell, Terrance Gray, F. Maxwell Harper, Haiyi Zhu. 2019

375 citations recorded by the source corpus.

Increasingly, algorithms are used to make important decisions across society. However, these algorithms are usually poorly understood, which can reduce transparency and evoke negative emotions. In this research, we seek to learn design principles for explanation interfaces that communicate how decision-making algorithms work, in order to help organizations explain their decisions to stakeholders, or to support users' "right to explanation". We conducted an online experiment where 199 participants used different explanation interfaces to understand an algorithm for making university admissions decisions. We measured users' objective and self-reported understanding of the algorithm. Our results show that both interactive explanations and "white-box" explanations (i.e. that show the inner workings of an algorithm) can improve users' comprehension. Although the interactive approach is more effective at improving comprehension, it comes with a trade-off of taking more time. Surprisingly, we also find that users' trust in algorithmic decisions is not affected by the explanation interface or their level of comprehension of the algorithm.

Subfields: Artificial Intelligence, Health Informatics, Safety Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

In Safety Research

Citations in the corpus

Expanding Explainability: Towards Social Transparency in AI systems (2021, background): “Only until recently have researchers began to conduct controlled lab studies to rigorously evaluate popular XAI techniques [12, 13, 15, 22, 27, 53, 80], as well as studies to understand real-world user needs for AI explainability [43, 50, 58].”
Bridging the Gap Between Ethics and Practice (2020, background): “propose designing explanations that consider user sentiments when explaining system decisions [21].”
The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practice (2023, background): “[51] Hao-Fei Cheng, Ruotong Wang, Zheng Zhang, Fiona O’Connell, Terrance Gray, F Maxwell Harper, and Haiyi Zhu.”
Are Explanations Helpful? A Comparative Study of the Effects of Explanations in AI-Assisted Decision-Making (2021, background): “In responding to this call, there is recently a growing line of literature on empirically evaluating the effectiveness of XAI methods (e.g., [14, 16, 41, 69, 71]).”
I Think I Get Your Point, AI! The Illusion of Explanatory Depth in Explainable AI (2021, methodology): “Prior work reported that providing users with interactive explanation facilities improved their subjective and objective model understanding [12].”
Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies (2021, methodology): “Model uncertainty Classification confidence (or probability) [10, 13, 21, 22, 43, 60, 68, 73, 86, 87, 144, 155], flag low confidence [90], uncertainty distribution (regression) [101] Local feature importance Coefficients [16, 28, 39, 43, 49, 55, 80, 81, 86, 87, 94, 106, 111, 125, 128, 143], attentio”
Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making (2020, background): “ople’s willingness to trust a ML system, whether by using a directly interpretable model, allowing user to inspect the model behavior, showing explanation or reducing the number of features presented [5, 16, 27, 29]. Many reasons could have contributed to this lack of effect. One is the complex mech”