Tell Me Where to Look: Guided Attention Inference Network

By Kunpeng Li, Ziyan Wu, Kuan-Chuan Peng, Jan Ernst, Yun Raymond Fu. 2018

568 citations recorded by the source corpus.

Weakly supervised learning with only coarse labels can obtain visual explanations of deep neural network such as attention maps by back-propagating gradients. These attention maps are then available as priors for tasks such as object localization and semantic segmentation. In one common framework we address three shortcomings of previous approaches in modeling such attention maps: We (1) make attention maps an explicit and natural component of the end-to-end training for the first time, (2) provide self-guidance directly on these maps by exploring supervision from the network itself to improve them, and (3) seamlessly bridge the gap between using weak and extra supervision if available. Despite its simplicity, experiments on the semantic segmentation task demonstrate the effectiveness of our methods. We clearly surpass the state-of-the-art on PASCAL VOC 2012 test and val. sets. Besides, the proposed framework provides a way not only explaining the focus of the learner but also feeding back with direct guidance towards specific tasks. Under mild assumptions our method can also be understood as a plug-in to existing weakly supervised learners to improve their generalization performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning Without Memorizing (2018, methodology): “As visual attention is also meaningful for segmentation (as shown in [11]), we intend to extend LwM to incremental segmentation in the near future.”
Image Super-Resolution Using Very Deep Residual Channel Attention Networks (2018, background): “Recently, tentative works have been proposed to apply attention into deep neural networks [23, 26, 27], ranging from localization and understanding in images [28, 29] to sequence-based networks [30, 31].”
Photographic Text-to-Image Synthesis with a Hierarchically-Nested Adversarial Network (2018, methodology): “Leveraging hierarchical representations of CNNs is an effective way to enhance implicit multi-scaling and ensembling for tasks such as image recognition [21, 48] and pixel or object classification [41, 25, 22, 37, 44, 50].”
Differentiable Patch Selection for Image Recognition (2021, background): “While this helps the model focus on relevant features [37, 50, 42, 26], become more interpretable [26], or include external data such as image captions [46, 2], the models will typically still process the whole image on a fixed input resolution.”
Attention mechanisms in computer vision: A survey (2021, unknown): “[97, 46, 139, 70, 128, 36, 11, 156] , [137, 143, 63, 161, 159, 6, 177, 80] , [67, 180, 10, 23, 24, 163, 155, 151] , [44, 49, 106, 64, 168, 18, 11, 36] , [48, 16, 169, 153, 174, 127, 53, 92] , [89, 21, 5, 134, 65, 171, 2, 109] , [45, 82, 157, 91, 100, 129, 146, 131] , [138, 85, 175, 76, 166, 133, 167”
Transformer Interpretability Beyond Attention Visualization (2020, methodology): “Being class-specific, and providing consistent results, this method is used by downstream applications, such as weakly-supervised semantic segmentation [21].”
Multi-class Token Transformer for Weakly Supervised Semantic Segmentation (2022, methodology): “A few methods developed heuristic strategies, such as “Hide & Seek” [31] and Erasing [40], applied either on images [24, 49] or feature maps [16,21] to drive the network to learn novel object patterns.”
Semi-Supervised Semantic Segmentation With Cross-Consistency Training (2020, background): “A significant number of approaches use a limited number pixel-level labels together with a larger number of inexact annotations, e.g ., region-level [48, 11] or image-level labels [32, 61, 54, 33].”
Single-Stage Semantic Segmentation From Image Labels (2020, background): “Consequently, higher-quality localisation [32] can further improve the segmentation accuracy.”
Discriminative Region Suppression for Weakly-Supervised Semantic Segmentation (2021, methodology): “One of the recent approaches is image-level and feature-level erasure of discriminative parts (Wei et al. 2017; Li et al. 2018; Hou et al. 2018).”
LayerCAM: Exploring Hierarchical Class Activation Maps for Localization (2021, background): “…ability of the class activation maps can make up such an issue of image-level labels, which further facilitates the ill-posed weakly-supervised tasks, such as weakly-supervised semantic segmentation [4], [5], [6], [7] and weakly-supervised object localization [8], [9] under image-level supervision.”
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems (2018, methodology): “For example, image segmentation maps (annotations of objects in images) are used to evaluate model generated saliency maps in weakly supervised object localization tasks [121].”