The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies

By Aniek F. Markus, Jan A. Kors, Peter R. Rijnbeek. Journal of Biomedical Informatics, 2020

686 citations recorded by the source corpus.

Artificial intelligence (AI) has huge potential to improve the health and well-being of people, but adoption in clinical practice is still limited. Lack of transparency is identified as one of the main barriers to implementation, as clinicians should be confident the AI system can be trusted. Explainable AI has the potential to overcome this issue and can be a step towards trustworthy AI. In this paper we review the recent literature to provide guidance to researchers and practitioners on the design of explainable AI systems for the health-care domain and contribute to formalization of the field of explainable AI. We argue the reason to demand explainability determines what should be explained as this determines the relative importance of the properties of explainability (i.e. interpretability and fidelity). Based on this, we propose a framework to guide the choice between classes of explainable AI methods (explainable modelling versus post-hoc explanation; model-based, attribution-based, or example-based explanations; global and local explanations). Furthermore, we find that quantitative evaluation metrics, which are important for objective standardized evaluation, are still lacking for some properties (e.g. clarity) and types of explanations (e.g. example-based methods). We conclude that explainable modelling can contribute to trustworthy AI, but the benefits of explainability still need to be proven in practice and complementary measures might be needed to create trustworthy AI in health care (e.g. reporting data quality, performing extensive (external) validation, and regulation).

Subfields: Artificial Intelligence, Health Informatics

View Paper: https://www.semanticscholar.org/paper/9ea8d646e832095bbc47776b34629ddd680ab641

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

Citations in the corpus

A Review of Taxonomies of Explainable Artificial Intelligence (XAI) Methods (2022, background): “Preventing discrimination [24, 41, 74], increasing trustworthiness [37, 41, 46], allocating responsibility [13, 41, 59], and generally, promoting human well-being [27, 46] is, in principle, possible with explainability – as long as researchers from different disciplines can come together to work on ”
Evaluating the Quality of Machine Learning Explanations: A Survey on Methods and Metrics (2021, methodology): “The ML explanation methods can also be divided into three types [23]:”
Explainable, trustworthy, and ethical machine learning for healthcare: A survey (2021, background): “Explainable ML can be a potential step towards trustworthy ML by building trust of clinicians in AI system [85].”
Explainable AI (XAI): A Systematic Meta-Survey of Current Challenges and Future Opportunities (2021, background): “While both terms are often used interchangeably in the literature, some examples from the selected papers distinguish them [20, 2, 21, 22, 23, 24].”
From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2022, background): “Additionally, besides saving time and resources [159], Herman [97] and Ancona et al.”
Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions (2023, background): “Furthermore, XAI has evolved into an exceedingly multidisciplinary, interdisciplinary, and transdisciplinary field.”