Explainable machine learning in deployment

By Umang Bhatt, Alice Xiang, Shubham Sharma, Adrian Weller, Ankur Taly, Yunhan Jia, Joydeep Ghosh, R. Puri, José M. F. Moura, Peter Eckersley. 2019

701 citations recorded by the source corpus.

Explainable machine learning offers the potential to provide stakeholders with insights into model behavior by using various methods such as feature importance scores, counterfactual explanations, or influential training data. Yet there is little understanding of how organizations use these methods in practice. This study explores how organizations view and use explainability for stakeholder consumption. We find that, currently, the majority of deployments are not for end users affected by the model but rather for machine learning engineers, who use explainability to debug the model itself. There is thus a gap between explainability in practice and the goal of transparency, since explanations primarily serve internal stakeholders rather than external ones. Our study synthesizes the limitations of current explainability techniques that hamper their use for end users. To facilitate end user interaction, we develop a framework for establishing clear goals for explainability. We end by discussing concerns raised regarding explainability.

Subfields: Artificial Intelligence, Management Information Systems, Safety Research, ML Systems & Hardware, AI Ethics, Fairness & Society

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Information Systems

In Safety Research

In ML Systems & Hardware

In AI Ethics, Fairness & Society

Citations in the corpus

Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, background): “[31] who conduct an explainability survey showing that explanations are often used only internally for troubleshooting rather than being shown to users.”
Algorithmic recourse under imperfect causal knowledge: a probabilistic approach (2020, background): “As machine learning algorithms are increasingly used to assist consequential decision making in a wide range of real-world settings [36, 41], providing explanations for the decision of these black-box models becomes crucial [7, 58].”
Getting a CLUE: A Method for Explaining Uncertainty Estimates (2020, methodology): “Using the same conﬁguration, we generate LIME and SHAP explanations for some MNIST digits to which our BNN assigns predictive entropy above our rejection threshold.”
Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims (2020, background): “, studying the efficacy of interpretability tools or exploring possible interfaces–would be welcome, as would further exploration of how practitioners currently use such tools [81] [82] [83] [78] [84].”
A survey of algorithmic recourse: definitions, formulations, solutions, and prospects (2020, background): “explanations citing their easy-to-understand nature [29, 30] and GDPR-compliance [177].”
Tree Space Prototypes: Another Look at Making Tree Ensembles Interpretable (2016, methodology): “Here, we compare to Shapley values because it is a state-of-the-art feature attribution method that satisfies several axiomatic guarantees [37], is popular in practice [1], and, as a feature-based explanation, is presented differently to users.”
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance (2020, background): “s, including whether the explanation improves users’ trust on the AI [34, 68] or enables users to simulate the model predictions [13, 55], or assisting developers to debug and train a better ML model [7,30]. , Vol. 1, No. 1, Article . Publication date: July 2020. 4 G. Bansal et al. Domain Task Perfo”
Explainable AI (XAI): Core Ideas, Techniques, and Solutions (2022, background): “[7] discuss how explanation schemes are adapted for ML engineers, end users, or other stakeholders.”
Counterfactual Fairness Is Basically Demographic Parity (2022, background): “One reason to prefer preserved orderings is that even soft guarantees for relative orderings of within group individuals leads to more transparent, trust-worthy and ultimately useful fair algorithms (Abdollahi and Nasraoui 2018; Bhatt et al. 2020).”
Explaining by Removing: A Unified Framework for Model Explanation (2020, background): “A view also mentioned by Bhatt et al. (2020), for example.”
You Shouldn't Trust Me: Learning Models Which Conceal Unfairness From Multiple Explanation Methods (2020, positive): “Since in practise these local saliency methods are the most popular approaches for evaluating model trustworthiness (Bhatt et al. 2019), it has been common to suggest that such methods can be used to inspect a model for fairness as follows.”
Post-hoc Interpretability for Neural NLP: A Survey (2021, background): “In contrast to other surveys and tutorials on interpretability methods [14, 15, 17, 25, 35, 79, 113, 119, 125] which only discusses the most popular approaches (usually 3 among input features , adversarial examples , influential examples , projection , and linguistic information ), this survey…”
Trustworthy Artificial Intelligence: A Review (2022, background): “There is a need to develop more explainability methods for external users who do not have any expertise to bridge the gap between transparency and actual implementation [17].”
A historical perspective of explainable Artificial Intelligence (2020, background): “As also pointed out in (Bhatt et al., 2020), the majority of deployments do not focus on the end-users, who are affected by the model, but rather on machine learning engineers, who use explainability to debug the model itself.”
Deep Neural Networks and Tabular Data: A Survey (2021, methodology): “With stricter regulations for automated decision-making [41] and the adoption of machine learning models in high-stakes domains such as ﬁnance and healthcare [45], [153], [154], interpretability is becoming a key concern.”
Problems with Shapley-value-based explanations as feature importance measures (2020, background): “Shapley-value-based explanations are primarily used for purposes of normative evaluation: deciding whether a model’s behavior is acceptable (Bhatt et al., 2020).”
Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty (2020, background): “Transparency includes a wide variety of efforts to provide stakeholders, such as model developers and end users, with relevant information about how a machine learning (ML) model works [16, 183].”
Evaluating and Aggregating Feature-based Model Explanations (2020, background): “These techniques are popular amongst machine learning scientists who want to sanity check a model before deploying it in the real world [ Bhatt et al. , 2020 ] .”