No Explainability without Accountability: An Empirical Study of Explanations and Feedback in Interactive ML

By Alison Smith-Renner, Ron Fan, M. Birchfield, Tongshuang Sherry Wu, Jordan L. Boyd-Graber, Daniel S. Weld, Leah Findlater. 2020

172 citations recorded by the source corpus.

Automatically generated explanations of how machine learning (ML) models reason can help users understand and accept them. However, explanations can have unintended consequences: promoting over-reliance or undermining trust. This paper investigates how explanations shape users' perceptions of ML models with or without the ability to provide feedback to them: (1) does revealing model flaws increase users' desire to "fix" them; (2) does providing explanations cause users to believe - wrongly - that models are introspective, and will thus improve over time. Through two controlled experiments - varying model quality - we show how the combination of explanations and user feedback impacted perceptions, such as frustration and expectations of model improvement. Explanations without opportunity for feedback were frustrating with a lower quality model, while interactions between explanation and feedback for the higher quality model suggest that detailed feedback should not be requested without explanation. Users expected model correction, regardless of whether they provided feedback or received explanations.

Subfields: Artificial Intelligence, Safety Research, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

In Safety, Robustness & Interpretability

Citations in the corpus

AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts (2021, background): “Human-AI interaction has been explored in domains such as classification [6, 59], drawing [24, 52], translation [28], creative writing [23, 27], and design ideation [36].”
Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI (2020, background): “al setup to evaluate unwarranted trust, and conclude that even data scientists are susceptible to develop unwarranted trust, despite some mathematical understanding of the models. Smith-Renner et al. [62] show similar conclusions on unwarranted trust in a different experimental setup. However, this ”
AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap (2023, background): “…without supporting control leaves users frustrated, while effective, efficient, and satisfying control cannot be achieved without transparency (Smith-Renner et al., 2020; Storms et al., 2022). cc While safety and control have become central topics in research and practices around LLMs (Keskar…”
Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies (2021, methodology): “Model uncertainty Classification confidence (or probability) [10, 13, 21, 22, 43, 60, 68, 73, 86, 87, 144, 155], flag low confidence [90], uncertainty distribution (regression) [101] Local feature importance Coefficients [16, 28, 39, 43, 49, 55, 80, 81, 86, 87, 94, 106, 111, 125, 128, 143], attentio”
Trust in AI: progress, challenges, and future directions (2024, unknown): “That is, the question is one of clarifying exactly what rights human stakeholders have, what obligations AI developers have, and how governments, developers, and watchdogs can enforce this scheme of rights and obligations (Blacklaws, 2018; Bovens et al., 2014; Smith-Renner et al., 2020).”
Expanding Explainability: Towards Social Transparency in AI systems (2021, background): “Moreover, while XAI is often claimed to be a critical step towards accountable AI, empirical studies have found little evidence that explanations improve a user’s perceived accountability or control over AI systems [81, 90].”
Explanation-Based Human Debugging of NLP Models: A Survey (2021, background): “data labels after being given explanations (Amershi et al., 2014; Smith-Renner et al., 2020).”