A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems

By Sina Mohseni, Niloofar Zarei, Eric D. Ragan. 2018

808 citations recorded by the source corpus.

The need for interpretable and accountable intelligent systems grows along with the prevalence of artificial intelligence ( AI ) applications used in everyday life. Explainable AI ( XAI ) systems are intended to self-explain the reasoning behind system decisions and predictions. Researchers from different disciplines work together to define, design, and evaluate explainable systems. However, scholars from different disciplines focus on different objectives and fairly independent topics of XAI research, which poses challenges for identifying appropriate design and evaluation methodology and consolidating knowledge across efforts. To this end, this article presents a survey and framework intended to share knowledge and experiences of XAI design and evaluation methods across multiple disciplines. Aiming to support diverse design goals and evaluation methods in XAI research, after a thorough review of XAI related papers in the fields of machine learning, visualization, and human-computer interaction, we present a categorization of XAI design goals and evaluation methods. Our categorization presents the mapping between design goals for different XAI user groups and their evaluation methods. From our findings, we develop a framework with step-by-step design guidelines paired with evaluation methods to close the iterative design and evaluation cycles in multidisciplinary XAI teams. Further, we provide summarized ready-to-use tables of evaluation methods and recommendations for different goals in XAI research.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety Research, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety Research

In Safety, Robustness & Interpretability

Citations in the corpus

Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, neutral): “Some attempts have been done recently around the measurement of XAI, as reviewed thoroughly in [349,350] .”
Trustworthy AI: From Principles to Practices (2021, background): “In the context of the psychology of users or the public, user studies or A/B test can provide a useful evaluation based on user satisfaction [249].”
Interpretable Machine Learning - A Brief History, State-of-the-Art and Challenges (2020, neutral): “Research-wise, the field is maturing in terms of methods surveys [75,41,120,96,1,6,23,15], further consolidation of terms and knowledge [42,22,82,97,88,17], and work about defining interpretability or evaluation of IML methods [74,73,95,49].”
Design Principles for Generative AI Applications (2024, background): “…guidelines focus on specifc types of AI technologies (e.g. text-to-image models [104]), specifc domains of use (e.g. creative writing [24]), or specifc issues regarding the use of AI, including ethics [11, 59, 69, 72], fairness [110], human rights [50], explainabil-ity [119], and user trust [186].”
From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2022, background): “Some surveys discuss evaluation as part of a broader review of XAI methods [1, 29, 32, 59, 81, 120, 170, 173, 175, 182, 303], or mainly discuss evaluation with user studies [52, 100, 136, 170].”
Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence (2023, background): “Overall, the interaction of explainability with AI highlights the importance of developing AI systems that are transparent, interpretable, and trustworthy in order to ensure their responsible and effective deployment in various domains [50–52].”
FUTURE-AI: international consensus guideline for trustworthy and deployable artificial intelligence in healthcare (2023, neutral): “72 The evaluations should also identify any limitations of the AI explanations (eg, they are clinically incoherent 73 or sensitive to noise or adversarial attacks, 74 they unreasonably increase the confidence in the AI generated results 75 ).”
User Perceptions and Trust of Explainable Machine Learning Fake News Detectors (2022, background): “For instance, expertise, typically resulting from knowledge and experience, helps users develop a more accurate understanding of AI’s working rationale and capabilities, which is a fundamental antecedent of trust in AI (Mohseni, Zarei, & Ragan, 2020; Mohseni et al., 2020).”