Artificial Intelligence and Black-Box Medical Decisions: Accuracy versus Explainability.

By Alex John London. 2019

706 citations recorded by the source corpus.

Although decision-making algorithms are not new to medicine, the availability of vast stores of medical data, gains in computing power, and breakthroughs in machine learning are accelerating the pace of their development, expanding the range of questions they can address, and increasing their predictive power. In many cases, however, the most powerful machine learning techniques purchase diagnostic or predictive accuracy at the expense of our ability to access "the knowledge within the machine." Without an explanation in terms of reasons or a rationale for particular decisions in individual cases, some commentators regard ceding medical decision-making to black box systems as contravening the profound moral responsibilities of clinicians. I argue, however, that opaque decisions are more common in medicine than critics realize. Moreover, as Aristotle noted over two millennia ago, when our knowledge of causal systems is incomplete and precarious-as it often is in medicine-the ability to explain how results are produced can be less important than the ability to produce such results and empirically verify their accuracy.

Subfields: Family Practice, General Health Professions, Health Informatics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Family Practice

In General Health Professions

In Health Informatics

Citations in the corpus

A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI (2019, background): “On the other hand, the exposition [163] argues that”
Artificial Intelligence (AI): Multidisciplinary perspectives on emerging challenges, opportunities, and agenda for research, practice and policy (2019, background): “Opportunities exists in conceptualising AI in the context of a black box approach where outputs should be judged using clinical trials and evidence-based practice to strive for accuracy (London, 2019).”
Redefining Radiology: A Review of Artificial Intelligence Integration in Medical Imaging (2023, background): “saliency maps, feature importance, and surrogate models, aid in visualising and explaining the rationale behind AI models’ decisions, making them accessible to both experts and lay audiences [99,100].”
Artificial Intelligence and Patient-Centered Decision-Making (2020, methodology): “It may happen, of course, that individual practitioners are not able to provide the relevant explanation themselves, but at least “ we expect that experts can marshal well-developed causal knowledge to explain their actions or recommendations ” (London 2019, p. 15).”
Unbox the black-box for the medical explainable AI via multi-modal and multi-centre data fusion: A mini-review, two showcases and beyond (2021, background): “Besides, the exposition argues [131] that a certain degree of opaqueness is appropriate, that is, it would be more important for us to deliver empirically checked reliable findings than to dwell too hard on how to unravel the black-box.”
Current Challenges and Future Opportunities for XAI in Machine Learning-Based Clinical Decision Support Systems: A Systematic Review (2021, background): “London [134] defends the ability to produce results and empirically verify their accuracy as more important than the ability to explain how such results are produced.”
On the ethics of algorithmic decision-making in healthcare (2019, background): “34 When prescribing medical treat-ment, clinicians often do not know why a certain drug works, as they lack an explanation of its biological mechanisms.”
Interpretability of machine learning‐based prediction models in healthcare (2020, methodology): “To obtain more information on a more general view of interpretability and explainable AI in healthcare, the reader is referred to works by Holzinger and colleagues (2019) or London (2019).”
Predicting cancer outcomes with radiomics and artificial intelligence in radiology (2021, background): “…models from the outset 155 , whereas other investigators contend that performance compared with present gold standard should be the most important metric to determine the usability of the imaging biomarker 156 , while others feel that there is a need to go even beyond explainable AI 157 .”