Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers

By Hila Chefer, Shir Gur, Lior Wolf. IEEE International Conference on Computer Vision, 2021

484 citations recorded by the source corpus.

Transformers are increasingly dominating multi-modal reasoning tasks, such as visual question answering, achieving state-of-the-art results thanks to their ability to contextualize information using the self-attention and co-attention mechanisms. These attention modules also play a role in other computer vision tasks including object detection and image segmentation. Unlike Transformers that only use self-attention, Transformers with co-attention require to consider multiple attention maps in parallel in order to highlight the information that is relevant to the prediction in the model’s input. In this work, we propose the first method to explain prediction by any Transformer-based architecture, including bi-modal Transformers and Transformers with co-attentions. We provide generic solutions and apply these to the three most commonly used of these architectures: (i) pure self-attention, (ii) self-attention combined with co-attention, and (iii) encoder-decoder attention. We show that our method is superior to all existing methods which are adapted from single modality explainability. Our code is available at: https://github.com/hila-chefer/Transformer-MM-Explainability.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/7ec5f207263100ea2d45db595712f611a74bafd9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Aggregating Nested Transformers (2021, background): “ViT is given high expectation to improve vision tasks beyond image classiﬁcation, with existing studies on generative modeling [11, 27, 28, 39], video understanding [2, 38], segmentation and detection [30, 34, 53], interpretability [1, 6, 7].”
ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension (2022, methodology): “Our usage of GradCAM follows Li et al. (2021), in which GradCAM is used to perform weakly supervised referring expression comprehension using the ALBEF model, and Chefer et al. (2021).”
Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understanding (2021, background): “…(Neimark et al. 2021; Akbari et al. 2021), segmentation and detection (Wang et al. 2021a; Liang et al. 2020; Kim et al. 2021), interpretability (Chefer, Gur, and Wolf 2021; Abnar and Zuidema 2020), a deeper understanding of the data efficiency and training difficulties from the architectural…”
Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond (2021, methodology): “As well, these ideas have also been used in Vision Transformers [48] for explaining image classification models [34, 185] or bi-modal transformer models [33].”
ConceptFusion: Open-set Multimodal 3D Mapping (2023, background): “They explore a different mechanism (attention-explainability [52]) to extract relevant regions corresponding to a text query; and demonstrate a complementary set of capabilities (completion of partly-observed objects, localizing hidden objects from language descriptions).”
Text2LIVE: Text-Driven Layered Image and Video Editing (2022, methodology): “Specifically, we use Chefer et al. [6] to automatically estimate a relevancy map R ( I s ) ∈ [0 , 1] 224 × 224 which roughly highlights the image regions that are most relevant to a given text T ROI .”
Doubly Right Object Recognition: A Why Prompt for Visual Rationales (2022, methodology): “In column 3, 5, 8, and 10, we show the saliency map [10] that models look to produce the prediction and rationales.”
WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentation (2023, methodology): “First, Trans-MM [5] is a recent model interpretation method applicable to Transformers that provides a pixel-level mask.”
Visual Classification via Description from Large Language Models (2022, methodology): “Chefer et al. (2021) extend similar techniques to vision transformers, such as those often used for vision-language models.”
LERF: Language Embedded Radiance Fields (2023, methodology): “Grad-CAM [31] and attention-based methods [7] provide a relevancy mapping between 2D images and text in vision-language models (e.g., CLIP).”
Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models (2023, background): “The extent to which attentioncanbeusedasanexplanationhasbeenwidelyexplored[Ab-nar and Zuidema 2020; Chefer et al. 2021, 2022b].”