What do we need to build explainable AI systems for the medical domain?

By Andreas Holzinger, Chris Biemann, Constantinos S. Pattichis, Douglas B. Kell. 2017

940 citations recorded by the source corpus.

Artificial intelligence (AI) generally and machine learning (ML) specifically demonstrate impressive practical success in many different application domains, e.g. in autonomous driving, speech recognition, or recommender systems. Deep learning approaches, trained on extremely large data sets or using reinforcement learning methods have even exceeded human performance in visual tasks, particularly on playing games such as Atari, or mastering the game of Go. Even in the medical domain there are remarkable results. The central problem of such models is that they are regarded as black-box models and even if we understand the underlying mathematical principles, they lack an explicit declarative knowledge representation, hence have difficulty in generating the underlying explanatory structures. This calls for systems enabling to make decisions transparent, understandable and explainable. A huge motivation for our approach are rising legal and privacy aspects. The new European General Data Protection Regulation entering into force on May 25th 2018, will make black-box approaches difficult to use in business. This does not imply a ban on automatic learning approaches or an obligation to explain everything all the time, however, there must be a possibility to make the results re-traceable on demand. In this paper we outline some of our research topics in the context of the relatively new area of explainable-AI with a focus on the application in medicine, which is a very special domain. This is due to the fact that medical professionals are working mostly with distributed heterogeneous and complex sources of data. In this paper we concentrate on three sources: images, *omics data and text. We argue that research in explainable-AI would generally help to facilitate the implementation of AI/ML in the medical domain, and specifically help to facilitate transparency and trust.

Subfields: Artificial Intelligence, Health Informatics, Medical & Health AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

In Medical & Health AI

Citations in the corpus

Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, neutral): “Unfortunately, scarce contributions have attempted to deﬁne such goals from a conceptual perspective [5,13,24,30] .”
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI) (2018, background): “Recently, researchers have conducted preliminary work aiming to make clinical AI-based systems explainable [37], [45]–[47].”
Explainable AI: The New 42? (2018, background): “The big advantage of such systems would include not only explainability, but deeper understanding and replicability [8].”
Explanation Methods in Deep Learning: Users, Values, Concerns and Challenges (2018, background): “The level of concern about these topics is reflected by official regulations such as the General Data Protection Regulation (GDPR), also mentioned in [14, 24], incentives to promote the field of explainability [22] and institutional initiatives to ensure the safe development of AI such as OpenAI.”
Learning Explainable Decision Rules via Maximum Satisfiability (2020, background): “The ability to explain the decision of a machine learning (ML) algorithm is a vital component of diagnostic, feedback, and humanin-the-loop systems [3]–[5].”
Explainable AI (XAI): Core Ideas, Techniques, and Solutions (2022, background): “It therefore follows that outputs from subsymbolic approaches when linked to background and contextual knowledge can generate user-speciﬁc and understandable explanations [30].”
AI in Medical Imaging Informatics: Current Challenges and Future Directions (2020, positive): “…multi-institutional collaboration is quickly becoming the vehicle driving the creation of well-curated and semantically annotated large cohorts that are further enhanced with research methods and results metadata, underpinning reproducible, extensible, and explainable research [59], [60].”
Explainable Sentiment Analysis with Applications in Medicine (2018, background): “[78], in which authors discussed the insight in developing explainable AI systems for supporting medical decisions.”
Trust in AI: progress, challenges, and future directions (2024, neutral): “…of transparency and explainability, meaning that models with the clearest explanations, such as decision trees, may not have a good performance, while those that are the most accurate, such as deep learning based models, are the least explainable (Agarwal et al., 2021; Holzinger et al., 2017).”
Explainable, trustworthy, and ethical machine learning for healthcare: A survey (2021, background): “of ML/DL models in any domain, especially in the medical domain [9].”
Unbox the black-box for the medical explainable AI via multi-modal and multi-centre data fusion: A mini-review, two showcases and beyond (2021, result): “As a result, several recent studies [120,123–127] have been devoted to the exploration of explainability in medical AI.”
Causability and explainability of artificial intelligence in medicine (2019, methodology): “Ante-hoc systems are interpretable by design towards glass-box approaches (Holzinger et al., 2017, 2018); typical examples include linear regression, decision trees and fuzzy inference systems.”
Deep learning in medical imaging and radiation therapy. (2018, background): “377 This, in part, has been driven by the European general data protection regulation that will go into effect in May 2018 and will make “ black-box ” approaches difficult to use in business.”
Pathologist-level interpretable whole-slide cancer diagnosis with deep learning (2019, background): “Despite its recognized human-level classification accuracy in relation to several diseases 5,6 , using deep learning as a diagnostic prediction mechanism is discouraged due to its lack of interpretability 14,15 .”
Machine Learning Interpretability: A Survey on Methods and Metrics (2019, background): “This does not imply outlawing automatic prediction systems or an obligation to explain everything all the time, but there must be a possibility to make the results re-traceable on demand [86], which may represent a challenging technical task [87].”