Questioning the AI: Informing Design Practices for Explainable AI User Experiences

By Q. Liao, D. Gruen, Sarah Miller. 2020

990 citations recorded by the source corpus.

A surge of interest in explainable AI (XAI) has led to a vast collection of algorithmic work on the topic. While many recognize the necessity to incorporate explainability features in AI systems, how to address real-world user needs for understanding AI remains an open question. By interviewing 20 UX and design practitioners working on various AI products, we seek to identify gaps between the current XAI algorithmic work and practices to create explainable AI products. To do so, we develop an algorithm-informed XAI question bank in which user needs for explainability are represented as prototypical questions users might ask about the AI, and use it as a study probe. Our work contributes insights into the design space of XAI, informs efforts to support design practices in this space, and identifies opportunities for future XAI work. We also provide an extended XAI question bank and discuss how it can be used for creating user-centered XAI.

Subfields: Artificial Intelligence, Information Systems and Management, Safety Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems and Management

In Safety Research

Citations in the corpus

Manipulating and Measuring Model Interpretability (2018, background): “[65] Q Vera Liao, Daniel Gruen, and Sarah Miller.”
Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies (2021, neutral): “theory [91, 103] and affinity diagramming [14, 64, 148, 149].”
Single-Turn Debate Does Not Help Humans Answer Hard Reading-Comprehension Questions (2022, background): “And though users want explanations of how models mark answers incorrect, most explanations that models output focus on the option selected (Liao et al., 2020).”
The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practice (2023, background): “Much of the effort branded as Explainable AI falls under this pursuit [e.g., 67, 111, 131 [128, 192, 211].”
Expanding Explainability: Towards Social Transparency in AI systems (2021, background): “whether current XAI techniques could appropriately enhance user trust [15, 80, 107] or the intended task performance, whether for decision making [12, 58, 111], model evaluation [6, 13, 22], or model”
Bridging the Gap Between Ethics and Practice (2020, background): “This controversial GDPR requirement is vague and difficult to satisfy in general, but international research efforts at developing Explainable AI (XAI) have blossomed [9, 47, 48, 106, 116, 75, 71].”
Exploring students’ perspectives on Generative AI-assisted academic writing (2024, positive): “…perspectives and demonstrate a range of explainable AI techniques and methodologies to better assist in sensemaking (Alqaraawi et al., 2020), enhancing user trust (Yang et al., 2020), or enabling shared decision making between student and AI system (Kim et al., 2024; Liao et al., 2020,).”
AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap (2023, background): “…cognitive science, and HCI, there is a long tradition of breaking down different types of explanations by their mechanism, stance, and the questions that they answer (e.g., what, how, why, why not, what if) (Graesser et al., 1996; Hilton, 1990; Keil, 2006; Liao et al., 2020; Lombrozo, 2012, 2016).”
How do Data Science Workers Collaborate? Roles, Workflows, and Tools (2020, background): “…emphasizes using various techniques (e.g., visualization [97]) and designs to make machine learning algorithms understandable by non-technical users [3, 21, 39, 57, 102], whereas we argue for the explanation of decisions among the various data science creators of machine learning algorithms.”
Current Challenges and Future Opportunities for XAI in Machine Learning-Based Clinical Decision Support Systems: A Systematic Review (2021, background): “XAI can also adapt usage or interaction behaviors to utilize the AI better [111].”
Human-Centered Explainable AI (XAI): From Algorithms to User Experiences (2021, methodology): “A mapping between categories of user questions in XAI question bank [63] and example XAI methods to answer these questions, with descriptions of their output in “Ways to explain” column.”
Design Principles for Generative AI Applications (2024, neutral): “For example, one set of papers examined requirements for explainable AI (XAI) and human-centered explainable AI (HCXAI) through experimental and heuristic methods [43, 98, 166], motivating “ex-plainability” as an important high-level concept.”
How Platform-User Power Relations Shape Algorithmic Accountability: A Case Study of Instant Loan Platforms and Financially Stressed Users in India (2022, background): “, of source-code, datasets, models, and processes surrounding the development of ADS) [21, 30, 55, 64, 83, 105] and explainable decisions (i.”
Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts (2023, methodology): “[29] and iterative dialog approaches by Lakkaraju et al.”