What Does Explainable AI Really Mean? A New Conceptualization of Perspectives

By Derek Doran, Sarah Schulz, Tarek R. Besold. 2017

483 citations recorded by the source corpus.

We characterize three notions of explainable AI that cut across research fields: opaque systems that offer no insight into its algo- rithmic mechanisms; interpretable systems where users can mathemat- ically analyze its algorithmic mechanisms; and comprehensible systems that emit symbols enabling user-driven explanations of how a conclusion is reached. The paper is motivated by a corpus analysis of NIPS, ACL, COGSCI, and ICCV/ECCV paper titles showing differences in how work on explainable AI is positioned in various fields. We close by introducing a fourth notion: truly explainable systems, where automated reasoning is central to output crafted explanations without requiring human post processing as final step of the generative process.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, neutral): “In this context, there is a broader distinction proposed by [24] discerning between 1) opaque systems, where the mappings from input to output are invisible to the user; 2) interpretable systems, in which users can mathematically analyze the mappings; and 3) comprehensible systems, in which the…”
Explanation Methods in Deep Learning: Users, Values, Concerns and Challenges (2018, methodology): “As stated by Doran et al. (2017), explanation methods may make predictive models such as DNNs more comprehensible.”
What do we need to build explainable AI systems for the medical domain? (2017, background): “Understanding, interpreting, explaining are often used synonymously in the context of explainable-AI (Doran et al., 2017), and various techniques of interpretation have been applied in the past.”
The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies (2020, background): “Terms are used differently in the public versus scientific setting [16] and across AI communities [21].”
Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey (2020, background): “and explainable AI are often generic and might be misleading [40] and should integrate some form of reasoning [41].”
Solving the Black Box Problem: A Normative Framework for Explainable Artificial Intelligence (2019, background): “1 Although this research program has already commanded significant attention (see, e.g., Burrell 2016; Doran et al. 2017; Lipton 2016; Ras et al. 2018; Zerilli et al. 2018), conceptual questions remain about the dual notions of “ opacity ” and “ transparency ” and, thus, about what it actually…”
Explainable AI (XAI): Core Ideas, Techniques, and Solutions (2022, background): “[13] postulate that for a system to be comprehensible (i.”
Four Responsibility Gaps with Artificial Intelligence: Why they Matter and How to Address them (2021, background): “Attention also went to the “accountability gap” in relation to autonomous weapon systems (AWS) (Heyns, 2013); (Meloni, 2016), and more generally within the discussion on explainability of algorithms and AI (Mittelstadt 2016; Doran et al. 2017; Pasquale, 2016).”
Explainable Sentiment Analysis with Applications in Medicine (2018, background): “al [58], most of the contributes in explainable AI field are focused on enabling explanations of models decisions, while few attempts are made to build models that “generate” explanations.”
Defining Explainable AI for Requirements Analysis (2018, unknown): “[7] corresponds to PostHoc Rationalisation in our vernacular.”
Towards Explainable Neural-Symbolic Visual Reasoning (2019, methodology): “In the model proposed by Doran et al. [2017], the black box, i.”
A review of research into automation in tourism: Launching the Annals of Tourism Research Curated Collection on Artificial Intelligence and Robotics in Tourism (2020, background): “(Brosnan, 1998); Uncanny Valley Theory (Mori, 2012) • ‘Explainable AI’ (Doran, Schulz, & Besold, 2017; Monroe, 2018) requirements for beneficial AI systems, and (3) developing systems that are robust against these requirements.”
Explainability for artificial intelligence in healthcare: a multidisciplinary perspective (2020, background): “Other terms such as interpretability and/or transparency are often used syn-onymously [2, 3].”
Semantic bottleneck for computer vision tasks (2018, background): “(explanation, justiﬁcation, transparency, etc. ) Several recent papers have tried to clarify those expressions [20,3,21,4,22,2,23,5] and separate the various approaches in two goals: build interpretable models and/or provide justiﬁ-cation of the prediction.”
A historical perspective of explainable Artificial Intelligence (2020, background): “Being able to provide an explanation of why a certain decision was made, has thus become a desirable property of intelligent systems (Doran, Schulz, & Besold, 2017).”
Algorithmic Bias in Education (2021, background): “Hence, progress in explainable artificial intelligence and interpretable artificial intelligence (Doran et al., 2018) and in research on explanation methods that make complex algorithms understandable (Zhou et al.”