What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use

By Sana Tonekaboni, Shalmali Joshi, Melissa D. McCradden, Anna Goldenberg. 2019

663 citations recorded by the source corpus.

Translating machine learning (ML) models effectively to clinical practice requires establishing clinicians' trust. Explainability, or the ability of an ML model to justify its outcomes and assist clinicians in rationalizing the model prediction, has been generally understood to be critical to establishing trust. However, the field suffers from the lack of concrete definitions for usable explanations in different settings. To identify specific aspects of explainability that may catalyze building trust in ML models, we surveyed clinicians from two distinct acute care specialties (Intenstive Care Unit and Emergency Department). We use their feedback to characterize when explainability helps to improve clinicians' trust in ML models. We further identify the classes of explanations that clinicians identified as most relevant and crucial for effective translation to clinical practice. Finally, we discern concrete metrics for rigorous evaluation of clinical explainability methods. By integrating perceptions of explainability between clinicians and ML researchers we hope to facilitate the endorsement and broader adoption and sustained use of ML systems in healthcare.

Subfields: Artificial Intelligence, Health Informatics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

Citations in the corpus

A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI (2019, background): “hers [43], [66], [137], [141] Comp. brain, neuro* [106], [157] lung [92], sleep [158] skin [159], other [136] Many more works have thus been dedicated to exploring explainability in the medical ﬁelds [11], [20], [43]. They provide summaries of previous works [21] including subﬁeldspeciﬁc reviews suc”
Explainability for artificial intelligence in healthcare: a multidisciplinary perspective (2020, background): “Given these considerations, explainability may be a key driver for the uptake of AI-driven CDSS in clinical practice, as trust in these systems is not yet established [22, 23].”
Towards Personalized Federated Learning (2021, background): “It is important for models in high stake applications such as healthcare to be explainable, where there is a strong need to justify decisions made [109].”
Unbox the black-box for the medical explainable AI via multi-modal and multi-centre data fusion: A mini-review, two showcases and beyond (2021, background): “As a result, several recent studies [81, 84, 85] have been devoted to the exploration of explainability in medical AI.”
The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies (2020, methodology): “For clinicians, this type of explanation is helpful to learn which features are responsible for the predicted outcome and to be able to compare with their own prior knowledge in limited time [3].”
Generative causal explanations of black-box classifiers (2020, background): “, [82]) and how technical tools can be brought to bear to address identified deficiencies.”
Current Challenges and Future Opportunities for XAI in Machine Learning-Based Clinical Decision Support Systems: A Systematic Review (2021, background): “Different clinicians will have different views, but all clinicians share a common ground—that of explainability through the eyes of patients [69].”
Explainable Artificial Intelligence Applications in Cyber Security: State-of-the-Art in Research (2022, background): “Despite simple models being frequently favored for their ease of explaining [91], these models’ explainability may be compromised in cases when highly engineered or heavy dimensional features are used [86].”
Assessing the (Un)Trustworthiness of Saliency Maps for Localizing Abnormalities in Medical Imaging (2020, background): “Explanation of an AI model should have enough transparency to confirm model outputs with medical knowledge and provide practical information that can be utilized by the clinicians to guide patient decision-making and management [34].”
Advancing Alzheimer Disease Prediction With Large Language Model–Based Linguistic Feature Analysis: Development and Validation Study (2026, unknown): “We evaluated the explanations using 5 key criteria derived from explainable AI literature [67-71], namely diagnostic connection, which assesses how well linguistic features are linked to AD diagnosis [67]; evidence support, which examines the concrete evidence provided from the transcript [68];…”
ECGTwin: Personalized ECG Generation Using Controllable Diffusion Model (2025, unknown): “The distribution and statistics of ECG record counts per patient are also shown in Table 4 and Figure 6.”