The false hope of current approaches to explainable artificial intelligence in health care.

By Marzyeh Ghassemi, Luke Oakden-Rayner, Andrew Beam. The Lancet Digital Health, 2021

1,184 citations recorded by the source corpus.

The black-box nature of current artificial intelligence (AI) has caused some to question whether AI must be explainable to be used in high-stakes scenarios such as medicine. It has been argued that explainable AI will engender trust with the health-care workforce, provide transparency into the AI decision making process, and potentially mitigate various kinds of bias. In this Viewpoint, we argue that this argument represents a false hope for explainable AI and that current explainability methods are unlikely to achieve these goals for patient-level decision support. We provide an overview of current explainability techniques and highlight how various failure cases can cause problems for decision making for individual patients. In the absence of suitable explainability methods, we advocate for rigorous internal and external validation of AI models as a more direct means of achieving the goals often associated with explainability, and we caution against having explainability be a requirement for clinically deployed models.

Subfields: Artificial Intelligence, Health Informatics

View Paper: https://www.semanticscholar.org/paper/634ed64dd2d4c53381fbcc53f4d0fa339711d799

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

Citations in the corpus

Human-Centered Explainable AI (XAI): From Algorithms to User Experiences (2021, background): “usage contexts, as many empirical studies failed to find conclusive evidence that adding XAI components improve performance for realistic AI-assisted user tasks such as judgment and decision making [15, 85, 103, 109] and image and video analysis [38, 41].”
Algorithm fairness in artificial intelligence for medicine and healthcare (2023, methodology): “Techniques such as Shapley additive explanations (SHAP) and Integrated Gradients have also found utility in explaining machine-learning predictions and dataset shift across a variety of applications 326–328 .”
Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions (2023, background): “Then, one possible course of action would be not to provide users with any XAI support if this is deemed detrimental or irrelevant in a given setting, for instance, in radiological settings (see [255,257]).”
Six Human-Centered Artificial Intelligence Grand Challenges (2023, background): “Explainability methods can be complex for end users and have a similar result of overtrust (Ghassemi et al., 2021; He et al., 2022).”
FUTURE-AI: international consensus guideline for trustworthy and deployable artificial intelligence in healthcare (2023, neutral): “68 Two recommendations for explainability are defined in the FUTURE-AI framework.”