Designing Theory-Driven User-Centric Explainable AI

By Danding Wang, Qian Yang, Ashraf Abdul, Brian Y. Lim. 2019

914 citations recorded by the source corpus.

From healthcare to criminal justice, artificial intelligence (AI) is increasingly supporting high-consequence human decisions. This has spurred the field of explainable AI (XAI). This paper seeks to strengthen empirical application-specific investigations of XAI by exploring theoretical underpinnings of human decision making, drawing from the fields of philosophy and psychology. In this paper, we propose a conceptual framework for building human-centered, decision-theory-driven XAI based on an extensive review across these fields. Drawing on this framework, we identify pathways along which human cognitive patterns drives needs for building XAI and how XAI can mitigate common cognitive biases. We then put this framework into practice by designing and implementing an explainable clinical diagnostic tool for intensive care phenotyping and conducting a co-design exercise with clinicians. Thereafter, we draw insights into how this framework bridges algorithm-generated explanations and human decision-making theories. Finally, we discuss implications for XAI design and development.

Subfields: Artificial Intelligence, Health Informatics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

Citations in the corpus

A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI (2019, methodology): “f AI. II. TYPES OF INTERPRETABILITY There has yet to be a widely-adopted standard to understand ML interpretability, though there have been works proposing frameworks for interpretability [10], [13], [35]. In fact, different works use different criteria, and they are justiﬁable in one Fig. 1. Overvi”
Manipulating and Measuring Model Interpretability (2018, background): “What is or is not interpretable must be defined by people’s behavior, not by what appeals to intuition [65, 78, 79, 110].”
Algorithmic management in a work context (2021, background): “In considering algorithmic management, different sets of concerns around explainability become salient, one of which is the varying technical literacies of workers who might interact with AI systems (Wang et al., 2019).”
Empowering educators to be AI-ready (2022, background): “The need for a less technical approach to educating people about AI has been highlighted, for example by researchers concerned with crowdsourcing and citizen science (see for example, Wang et al., 2019; Smith et al., 2020;Yang et al., 2018) and explainable AI (Liao et al., 2020).”
How Platform-User Power Relations Shape Algorithmic Accountability: A Case Study of Instant Loan Platforms and Financially Stressed Users in India (2022, background): “, to help the users make informed choices when interacting with ADS) [10, 21, 42, 83, 100, 119]; or be embedded in other mechanisms such as algorithmic audits [26, 46, 112] and impact assessments [101, 102].”
Bridging the Gap Between Ethics and Practice (2020, background): “This controversial GDPR requirement is vague and difficult to satisfy in general, but international research efforts at developing Explainable AI (XAI) have blossomed [9, 47, 48, 106, 116, 75, 71].”
Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems (2020, background): “Even though conceptual frameworks that consider cognitive processes in decision-making with DSS have been proposed recently [41], further research is needed to thoroughly investigate how to incorporate DSS into human decision-making and the eect of cognitive processes while making system-assisted d”
Expanding Explainability: Towards Social Transparency in AI systems (2021, background): “reviewed decision-making theories and identified many gaps in XAI output to support the complete cognitive processes of human reasoning [101].”
"Hello AI": Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making (2019, background): “Domains of study include recommender systems [4], medicine [12, 75], social media [22], creativity [14], and advertisements [23].”
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance (2020, methodology): “’s suggestion to mitigate heuristic biases by explaining “multiple outcomes” [74].”
Explanations Can Reduce Overreliance on AI Systems During Decision-Making (2022, background): “Potential cognitive mechanisms that underlie these results suggest that the mere presence of an explanation increases trust [96, 97], and that explanations anchor humans to the prediction [9, 91].”
Evaluating the Quality of Learning Resources: A Learnersourcing Approach (2021, background): “While some studies suggest that explainable AI (XIA) [67] is not always required [68], the use of black-box outcomes seems to be inadequate for educational settings where educators strive to enable students to develop their own vision, reasoning, and appreciation for inquiry and investigation, and…”
Human-Centered Explainable AI (XAI): From Algorithms to User Experiences (2021, background): “A research community of human-centered XAI [33, 35, 99] has emerged, which bring in cognitive, sociotechnical, design perspectives, and more.”
Questioning the AI: Informing Design Practices for Explainable AI User Experiences (2020, methodology): “To close the gap between XAI algorithms and user needs for effective transparency, the HCI community has called for interdisciplinary collaboration [4] and user-centered approaches to explainability [92].”