Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)

By Been Kim, Martin Wattenberg, Justin Gilmer, Carrie J. Cai, James Wexler, Fernanda B. Viégas, Rory Sayres. 2017

2,408 citations recorded by the source corpus.

The interpretation of deep learning models is a challenge due to their size, complexity, and often opaque internal state. In addition, many systems, such as image classifiers, operate on low-level features rather than high-level concepts. To address these challenges, we introduce Concept Activation Vectors (CAVs), which provide an interpretation of a neural net's internal state in terms of human-friendly concepts. The key idea is to view the high-dimensional internal state of a neural net as an aid, not an obstacle. We show how to use CAVs as part of a technique, Testing with CAVs (TCAV), that uses directional derivatives to quantify the degree to which a user-defined concept is important to a classification result--for example, how sensitive a prediction of "zebra" is to the presence of stripes. Using the domain of image classification as a testing ground, we describe how CAVs may be used to explore hypotheses and generate insights for a standard image classification network as well as a medical application.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Model Cards for Model Reporting (2018, background): “be complete or exhaustive, and may be tailored depending on the model, context, and stakeholders. Additional details may include, for example, interpretability approaches, such as saliency maps, TCAV [33], and Path-Integrated Gradients [38, 43]); stakeholder-relevant explanations (e.g., informed by ”
Scalable agent alignment via reward modeling: a research direction (2018, background): “…attend to when they make decisions (Greydanus et al., 2018), evaluating models’ sensitivity to the presence/intensity of high-level human concepts (Kim et al., 2017), optimizing a model to be more interpretable with humans in the loop (Lage et al., 2018), translating neural activations into…”
CausaLM: Causal Model Explanation Through Counterfactual Language Models (2020, background): “Moreover, there’s been, to the best of our knowledge, no work on measuring the effect of concepts on models’ predictions in NLP (see Kim et al. (2018) and Goyal et al. (2019a) for a discussion in the context of computer vision).”
Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, neutral): “• Some methods [82,144] are classiﬁed on a single category ( Explanation by simpliﬁcation for Multi-Layer Neural Network ) in Fig.”
Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, background): “Even if we create a vector in the latent space that is aimed towards a single concept (as is done in Kim et al., 2018; Zhou et al., 2018b), that vector could activate highly on multiple concepts, which means the signal for the two concepts is not disentangled.”
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI (2019, methodology): “r studying the extent of effect x ihas on a decision made by a deep NN [129]. Challenges and Future Prospects. There seems to be a concern with locality and globality of the concepts. As mentioned in [95], to achieve global quantiﬁcation for interpretability, explanation must be given for a set of e”
Generative Counterfactual Introspection for Explainable Deep Learning (2019, methodology): “To understand how components of the network work, a variety of the methods have been introduced to visualization the feature (or pattern) the given neuron or layer aim to capture [16]–[18] or examining the representation of the high-level concept in the latent representations [19].”
EDUCE: Explaining model Decisions through Unsupervised Concepts Extraction (2019, methodology): “[11] propose to explain a model’s prediction by learning concept activation vectors.”
Neural Prototype Trees for Interpretable Fine-grained Image Recognition (2020, neutral): “activation maximization [39, 41] to visualize neurons, deconvolution or upconvolution [12, 54] to explain layers, generating image exemplars [18] to explain the latent space, or concept activation vectors [26] to explain model sensitivity.”
Concept whitening for interpretable image recognition (2020, methodology): “In detail, (1) we check the most activated images on these axes, (2) we look at how images are distributed in a two-dimensional (2D) slice of the latent space and (3) we show how realizations of the same concept change if we apply CW on different layers.”
Explainable machine learning in deployment (2019, neutral): “These techniques attempt to characterize the concepts learned by the model [31], simpler models learned from the representation of complex models [17], prototypical samples from a particular model output [10], or the topology of the data itself [20].”
Interpretability Beyond Classification Output: Semantic Bottleneck Networks (2019, neutral): “The first being post-hoc interpretation, for which we take an already trained and well performing model and dissect its decisions a-posteriori to identify important input features via attribution [2, 21, 13, 33, 22, 24] or attempt to assign meaning to single or groups of features in order to underst”
Interpretable Basis Decomposition for Visual Explanation (2018, background): “Concurrent work [14] proposes examining the behavior of representations in the direction of a set of semantic concept vectors learned from a pre-defined dataset.”
Towards Robust Interpretability with Self-Explaining Neural Networks (2018, background): “Interpretable concept learning is a challenging task in its own right [8], and as other aspects of interpretability, remains ill-defined.”
Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? (2020, neutral): “(8)Also referred to as feature-attribution explanations (Kim et al., 2017).”
The challenge of crafting intelligible intelligence (2018, methodology): “• Changing the vocabulary by adding (or removing) a feature in the explanatory model, either from a predefined set, by using methods from machine teaching, or with concept activation vectors.(17) • Perturbing the input example to see the effect on both prediction and explanation.”
Explanations can be manipulated and geometry is to blame (2019, background): “[18] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres.”