Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance

By Gagan Bansal, Tongshuang Sherry Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, Daniel S. Weld. 2020

965 citations recorded by the source corpus.

Many researchers motivate explainable AI with studies showing that human-AI team performance on decision-making tasks improves when the AI explains its recommendations. However, prior studies observed improvements from explanations only when the AI, alone, outperformed both the human and the best team. Can explanations help lead to complementary performance, where team accuracy is higher than either the human or the AI working solo? We conduct mixed-method user studies on three datasets, where an AI with accuracy comparable to humans helps participants solve a task (explaining itself in some conditions). While we observed complementary improvements from AI augmentation, they were not increased by explanations. Rather, explanations increased the chance that humans will accept the AI’s recommendation, regardless of its correctness. Our result poses new challenges for human-centered AI: Can we develop explanatory approaches that encourage appropriate trust in AI, and therefore help generate (or improve) complementary performance?

Subfields: Artificial Intelligence, Management Information Systems, Management Science and Operations Research, Safety, Robustness & Interpretability, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Information Systems

In Management Science and Operations Research

In Safety, Robustness & Interpretability

In Human-Computer Interaction

Citations in the corpus

Measuring Progress on Scalable Oversight for Large Language Models (2022, background): “Especially relevant from this literature is Bansal et al. (2021), which focuses on the human–AI collaboration in the critical regime where (i) we can reasonably expect a human–AI team to outperform the AI system and (ii) the AI system can offer explanations.”
AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts (2021, background): “Human-AI interaction has been explored in domains such as classification [7, 62], drawing [26, 54], translation [30], creative writing [25, 29], and design ideation [38].”
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models (2021, background): “9 In our case, the concern might be that users can develop an over-reliance on Polyjuice (Bansal et al., 2021) and hastily accept its generations.”
To Trust or to Think (2021, background): “Indeed, some studies demonstrate that explanations are interpreted as a general signal of competence—rather than being evaluated individually for their content—and just by their presence can increase the trust in and overreliance on the AI [2].”
Human-Centered Explainable AI (XAI): From Algorithms to User Experiences (2021, background): “For example, recent HCI studies repeatedly found that showing uncertainty information of individual predictions is more effective than local explanations to help people form appropriate reliance on AI [14, 105].”
Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Making (2021, background): “, in accuracy; also known as complementary performance [2]).”
Human–AI collaboration enables more empathic conversations in text-based peer-to-peer mental health support (2022, background): “Collaboration between humans and AI can take many forms, but specific formulations of human–AI collaboration remain poorly defined and challenging to measure 11 .”
Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies (2021, methodology): “, convolution neural networks [3, 27], recurrent neural networks [26, 60], BERT [80], and RoBERTa [13], a hybrid LSTM and CNN model for VQA task [115].”
Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions (2023, background): “Thus, whether information biases the generative process, as, for example, with the prompt in a language-to-image stable diffusion model, is sufficiently original for intellectual property and author attribution claims remains an open question.”
Using Interactive Feedback to Improve the Accuracy and Explainability of Question Answering Systems Post-Deployment (2022, methodology): “5We follow Berg-Kirkpatrick et al. (2012) to conduct the statistical significant test and COMBINEDRERANKER is a much stronger360 model than any of the models, indicating that learn-361 ing signals presented in feedback data and the pre-362 deployment data are complementary to each other.363…”
Explanations Can Reduce Overreliance on AI Systems During Decision-Making (2022, background): “Of the possible error types in human-AI decision-making (Figure 1), overreliance is the most frequent outcome in empirical AI studies [4, 8, 9, 53, 97].”
Natural Language Deduction through Search over Statement Compositions (2022, background): “This limitation prevents users from understanding and accommodating models’ affordances (Hase and Bansal, 2020; Bansal et al., 2021).”
Are Explanations Helpful? A Comparative Study of the Effects of Explanations in AI-Assisted Decision-Making (2021, background): “Finally, Desideratum 3 concerns the ultimate goal of AI-assisted decision-making, that is, to maximize the joint human-AI team performance [5, 6, 11].”
Single-Turn Debate Does Not Help Humans Answer Hard Reading-Comprehension Questions (2022, methodology): “However, Bansal et al. (2021) highlight that these studies use models that outperform humans at the task in question, undermining the motivation for providing a model’s explanation alongside its prediction.”
Two-Turn Debate Doesn't Help Humans Answer Hard Reading Comprehension Questions (2022, background): “Previous studies have reported that model-generated explanations can aid humans in some tasks (Cai et al., 2019; Lundberg et al., 2018; Schmidt and Biessmann, 2019; Lai and Tan, 2019), though only when the models are generally able to outperform humans at that task (Bansal et al., 2021).”
AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap (2023, background): “…a wave of HCI studies repeatedly showed that AI explanations can lead to overreliance—increasing people’s tendency to mistakenly follow the AI outputs even when they are wrong (Bansal et al., 2021; V. Chen et al., 2023; Poursabzi-Sangdeh et al., 2021; X. Wang and Yin, 2021; Y. Zhang et al., 2020).”
Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphs (2020, background): “Their limitation is that they can be used to persuade users that the model is reliable when it is not (Bansal et al., 2020)—an ethical issue raised by Herman (2017).”
Harnessing the Wisdom of LLM Crowds through Complementarity-Driven Iterative Collaboration (2026, unknown): “Building upon the established definitions of complementarity in the literature (Bansal et al. 2021, Donahue et al. 2022), we define the complementarity of workers in our sequential collaboration scenario.”