Transformer Interpretability Beyond Attention Visualization

By Hila Chefer, Shir Gur, Lior Wolf. 2020

1,103 citations recorded by the source corpus.

Self-attention techniques, and specifically Transformers, are dominating the field of text processing and are becoming increasingly popular in computer vision classification tasks. In order to visualize the parts of the image that led to a certain classification, existing methods either rely on the obtained attention maps or employ heuristic propagation along the attention graph. In this work, we propose a novel way to compute relevancy for Transformer networks. The method assigns local relevance based on the Deep Taylor Decomposition principle and then propagates these relevancy scores through the layers. This propagation involves attention layers and skip connections, which challenge existing methods. Our solution is based on a specific formulation that is shown to maintain the total relevancy across layers. We benchmark our method on very recent visual Transformer networks, as well as on a text classification problem, and demonstrate a clear advantage over the existing explainability methods. Our code is available at: https://github.com/hila-chefer/Transformer-Explainability.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Foundation Posteriors for Approximate Probabilistic Inference (2022, neutral): “Prior works [6, 60, 1, 14] using transformer networks in natural language have re-purposed the”
Aggregating Nested Transformers (2021, methodology): “ViT is given high expectation to improve vision tasks beyond image classiﬁcation, with existing studies on generative modeling [11, 27, 28, 39], video understanding [2, 38], segmentation and detection [30, 34, 53], interpretability [1, 6, 7].”
CLIP-ReID: Exploiting Vision-Language Model for Image Re-Identification without Concrete Text Labels (2022, methodology): “Finally, we perform visualization experiments using the (Chefer, Gur, and Wolf 2021) method to show the focused areas of the model.”
Explainability for Large Language Models: A Survey (2023, methodology): “Both methods have been applied to derive the relevance score of inputs in Transformer-based models (Wu & Ong, 2021; Chefer et al., 2021).”
DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification (2021, methodology): “(c) visualizes the impact of each spatial location on the ﬁnal prediction in the DeiT-S model [25] using the visualization method proposed in [3].”
Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers (2021, methodology): “Since each such map is comprised of h heads, we follow [5] and use gradients to average across heads.”
A Survey on Vision Transformer (2020, background): “The current literature usually analyzes the effect in an intuitive way [55], [23].”
CLIP on Wheels: Zero-Shot Object Navigation as Object Localization and Exploration (2022, methodology): “For transformer-based architectures (i.e., ViT [22]), we adopt the modifications developed by Chefer et al. [13].”
Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond (2021, methodology): “As well, these ideas have also been used in Vision Transformers [48] for explaining image classification models [34, 185] or bi-modal transformer models [33].”
A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, methodology): “The study [370] introduced an innovative approach for assessing relevancy in Transformer networks in the computer vision domain.”
Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content? (2022, methodology): “Combining Q16 with explainable AI methods such as [Chefer et al. 2021] to explain the reasons is likely to improve the datasheet. mean rating < 2 .”
A foundation model for generalizable disease detection from retinal images (2023, background): “All studies must disclose on these points even when the disclosure is negative.”
Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer (2021, background): “Especially in deeper layers ( e.g. , layer 10), the class token usually augments informative tokens with higher attention scores and has a sparse attention response, which is quite consistent to the visualization result provided by (Chefer, Gur, and Wolf 2021) for transformer interpretability.”
TransPose: Keypoint Localization via Transformer (2020, methodology): “We also notice a recent paper [10] that develops LRPbased [2] method to compute relevance to explain the predictions of Transformer.”
IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformers (2021, background): “On the other hand, previous works [6, 8] have already shown the vulnerable interpretability of the original vision transformer, where the raw attention comes from the architecture sometimes fails to perceive the informative region of the input images.”
CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation (2022, methodology): “Third, we modify an interpretability method [13, 70] designed to extract object relevancy from vision transformers (ViTs) [24].”