Explainable AI: Beware of Inmates Running the Asylum Or: How I Learnt to Stop Worrying and Love the Social and Behavioural Sciences

By Tim Miller, Piers Douglas Lionel Howe, Liz Sonenberg. 2017

428 citations recorded by the source corpus.

In his seminal book `The Inmates are Running the Asylum: Why High-Tech Products Drive Us Crazy And How To Restore The Sanity' [2004, Sams Indianapolis, IN, USA], Alan Cooper argues that a major reason why software is often poorly designed (from a user perspective) is that programmers are in charge of design decisions, rather than interaction designers. As a result, programmers design software for themselves, rather than for their target audience, a phenomenon he refers to as the `inmates running the asylum'. This paper argues that explainable AI risks a similar fate. While the re-emergence of explainable AI is positive, this paper argues most of us as AI researchers are building explanatory agents for ourselves, rather than for the intended users. But explainable AI is more likely to succeed if researchers and practitioners understand, adopt, implement, and improve models from the vast and valuable bodies of research in philosophy, psychology, and cognitive science, and if evaluation of these models is focused more on people than on technology. From a light scan of literature, we demonstrate that there is considerable scope to infuse more results from the social and behavioural sciences into explainable AI, and present some key results from these fields that are relevant to explainable AI.

Subfields: Artificial Intelligence, Health Informatics, Safety Research, Safety, Robustness & Interpretability, AI Ethics, Fairness & Society

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

In Safety Research

In Safety, Robustness & Interpretability

In AI Ethics, Fairness & Society

Citations in the corpus

Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, neutral): “• Accessibility: A minor subset of the reviewed contributions argues for explainability as the property that allows end users to get more involved in the process of improving and developing a certain ML model [37,86] .”
Counterfactuals in Explainable Artificial Intelligence (XAI): Evidence from Human Reasoning (2019, background): “However, to maximize their effectiveness, it will be useful for XAI to incorporate information from psychological experiments about the way people create and comprehend counterfactuals, for counterfactuals of different structure and content, and with various relations.”
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI) (2018, methodology): “A call for using social science models in XAI was made in [155].”
What do we need to build explainable AI systems for the medical domain? (2017, background): “Explainable-AI calls for conﬁdence, safety, security, privacy, ethics, fairness and trust (Kieseberg et al., 2016), and puts usability on the research agenda, too Miller et al. (2017).”
Explanation in Artificial Intelligence: Insights from the Social Sciences (2017, background): “[132] shows in a small sample that research in explainable AI typically does not cite or build on frameworks of explanation from social science.”
Manipulating and Measuring Model Interpretability (2018, background): “[79] Tim Miller, Piers Howe, and Liz Sonenberg.”
Explaining Explanations in AI (2018, background): “ity of currently available methods will, at best, produce locally reliable but globally 2The dichotomy between explanations in machine learning and elsewhere is well illustrated by the recent work by Miller et al. (2017) who found that none of the papers on a curated reading list for xAI made use of”
Interpreting Interpretability: Understanding Data Scientists' Use of Interpretability Tools for Machine Learning (2020, background): “To that end, Miller [47, 48] and Lombrozo [41] propose simplicity, generality, and coherence as the main evaluation criteria for explanations.”
Current Challenges and Future Opportunities for XAI in Machine Learning-Based Clinical Decision Support Systems: A Systematic Review (2021, background): “Models should be built in collaboration with input from those with expertise from the ﬁelds of social and behavioral science, philosophy, psychology, and cognitive science [19,20,64].”
Explaining by Removing: A Unified Framework for Model Explanation (2020, background): “This makes the interpretability literature difficult to navigate and raises questions about whether existing methods relate to human processes for explaining complex decisions (Miller et al., 2017; Miller, 2019).”
Human-level explanatory biases for person re-identification (2018, background): “Despite the remarkable progress in deep learning in recent years, a major challenge for present systems is to generate explanations compelling enough to serve as useful accounts of the system’s operations [1].”
A Review of Deep Learning Models for Time Series Prediction (2019, neutral): “For instance, IJCAI specially organized a workshop discussing Explainable Arti-ﬁcial Intelligence (XAI) from 2017 [218], [219].”
Defining Explainable AI for Requirements Analysis (2018, positive): “[18] discuss why it is important to “beware of the inmates running the asylum”, noting that explanations should be targeted towards the right user audience—who are not necessarily the AI practitioner, software engineer, or even the domain expert.”
The importance of interpretability and visualization in machine learning for applications in medicine and health care (2019, background): “This concern has been expressively stated as the risk of “inmates running the asylum” [24].”